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Striving Readers 
Final Evaluation Report: 
Danville, Kentucky 


Executive Summary of Findings: Implementation and Impact 


This Striving Readers evaluation examined the impact of a targeted intervention for 
struggling adolescent readers and investigated the impact of a whole-school literacy model on 
students in participating schools. This study was conducted in ten middle, nine high, and two 6-12 
schools in seven rural school districts serving large percentages of at-risk students in Kentucky. 
The targeted intervention for struggling readers was the Learning Strategies Curriculum (LSC) 
developed by the University of Kansas Center for Research on Learning. The whole-school 
intervention was the Collaborative for Teaching and Learning Adolescent Literacy Model (ALM). 


The LSC intervention was evaluated through a multiple cohort within-school randomized 
field trial in which new students were assigned to treatment and control groups each year of the 
project. The LSC was a supplement to the regular curriculum wherein students in the targeted 
intervention participated in a reading class in place of an elective as part of their regular school 
day. The control-group condition was “business as usual,” wherein students in the control group 
take a regular elective such as band, theater arts, civics, or physical education. This study 
examined the impact of the LSC on struggling sixth- and ninth-grade students’ reading 
achievement, strategy use, and motivation. 


In addition to the targeted intervention, this study evaluated the impact of ALM on teacher 
efficacy and student achievement in reading and writing. This school-wide model provided 
professional development for content teachers in integrating literacy strategies across content 
areas in service of content learning. All teachers in the Striving Readers schools were expected to 
implement the whole-school intervention, and all students were expected to receive instruction in 
the whole-school techniques across the content areas. 


In this Striving Readers project, each school employed a literacy coach who was responsible 
for both teaching the targeted intervention to struggling readers and for coaching content 
teachers in implementing the whole-school intervention’. Through the project, these coaches had 
the opportunity to participate in a literacy leadership certification program through the University 
of Louisville (U of L). 


*When describing the intervention teacher’s role, we will refer to the literacy coach as the “LSC teacher,” 
and when describing the whole-school program coach’s role, we will refer to the literacy coach as the “ALM coach”. 
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The impact research questions that motivated the study design and analysis plan are: 


e What is the impact of the LSC on the reading achievement, strategy use, and 
motivation of struggling adolescent readers? 


e What changes occurred in the personal and collective efficacy of the LSC teachers/ALM 
coaches over the course of the project? 


e What is the impact of the ALM on student achievement in reading and the content 
areas? 


e What is the impact of the ALM on content teachers’ personal and collective efficacy for 
literacy teaching? 


LSC Targeted Intervention 


This study examined both the implementation and impact of the LSC over the course of the 
4-year Striving Readers project. 


Implementation. During each year of the project, LSC teachers participated in training and 
on-site support, and training was provided for school administrators. In year 1, LSC teachers were 
provided 7 days professional development training and up to 60 hours support from visits by 
mentor coaches. In year 2, the LSC teachers were provided 7.5 days training and up to 48 hours of 
on-site support. In year 3, LSC teachers were provided 7 days professional development training 
and up to 72 hours of on-site support. In year 4, LSC teachers were provided 4.5 days professional 
development training and up to 60 hours of on-site support. School administrators were provided 
a total of 6.5 days training in the intervention for the length of the grant. Overall, participation in 
the professional development inputs for middle, high, and 6-12 LSC teachers and administrators 
was high, with 100% of teachers participating fully in the training each year of the project. 


Classroom implementation fidelity was measured through classroom observations and by 
calculating the percentage of days that students received instruction by a trained LSC teacher 
(teacher attendance). Observations indicated that fidelity to the targeted intervention was 
relatively low in year 1, with sixth-grade teachers achieving 58.5% implementation fidelity and 
ninth-grade teachers achieving 70.4% fidelity. In year 2, implementation fidelity increased to 80% 
for sixth-grade teachers and 78.5% for ninth-grade teachers. In year 3, implementation fidelity 
continued to rise, with sixth-grade teachers achieving 87.2% fidelity and ninth-grade teachers 
achieving 86.6% fidelity of implementation. In the final year of the grant, sixth-grade teachers 
rose to 87.7%, and ninth-grade teachers decreased slightly to 82.9%. Teacher attendance was 
relatively high across all years of the project. 


Impacts. This study used hierarchical linear modeling (HLM) analyses to measure the 
impacts of the LSC on sixth-grade (middle school) and ninth-grade (high school) students’ reading 
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achievement, strategy use, and motivation. In this Striving Readers study, there were no impacts 
on sixth-grade students’ reading achievement, as measured by the Group Reading Assessment and 
Diagnostic Evaluation, but there were significant impacts for ninth-grade students’ achievement. 
Results on the Metacognitive Awareness of Reading Strategies Inventory (Mokhtari & Reichard, 
2002) indicated marginally significant impacts for sixth grade on students’ self-reported reading 
strategy use but revealed no significant impacts on strategy use for ninth grade. Results on the 
Motivation to Read Questionnaire (Wigfield & Guthrie, 1997) indicated significant effects for both 
sixth- and ninth-grade students in the area of reading motivation. 


ALM Whole-School Intervention 


In addition to the targeted intervention, this Striving Readers study examined the 
implementation of a school-wide model, ALM, its impacts on content teachers’ personal and 
collective efficacy for literacy teaching, and its impacts on student achievement in reading. 


Implementation. The professional development inputs for the whole-school intervention 
were multifaceted, as training was provided for content teachers, ALM coaches, and school 
administrators. Over the 4 years of the project, content teachers were provided at least 11 days 
training in the whole-school model, and teachers’ participation in those training sessions (as 
measured by attendance) was relatively high. Additional ongoing, job-embedded professional 
development for content teachers was provided by the school-based ALM coaches. The ALM 
coaches were provided 37 days of training in the school-wide intervention, and their participation 
was high in all years, overall, with 100% of ALM coaches participating fully in the training each 
year of the project. To further their professional development, ALM coaches were provided up to 
246 hours of on-site support by a mentor coach over the course of the 4 years. School 
administrators were provided 14 days professional development in the school-wide model over 
the 4 years, and their participation (as measured by attendance) was relatively high in all years. 


Evaluators conducted classroom observations to assess content teachers’ classroom 
implementation of the whole-school model. In year 1, all English/Language Arts (E/LA) teachers 
were observed, and in year 2, E/LA teachers and a sample of teachers in other content areas were 
observed. In years 3 and 4, a random sample of teachers was observed across content areas. In 
general, middle school teachers implemented the ALM at higher levels than high school teachers. 


Impacts. The impacts of the whole-school model were assessed in terms of teacher 
efficacy and student achievement. Teacher efficacy was measured through a teacher efficacy 
survey that was administered to ALM coaches/LSC teachers, all content teachers in Striving 
Readers schools, and content teachers in matched schools. All teachers in Striving Readers 
schools and teachers in matched schools completed a teacher efficacy survey at the start of the 
project and again at the end of each year of the project. Results indicated that teachers’ personal 
efficacy at Striving Readers schools was lower than at matched schools at the start of the project 
and increased steadily. Teachers’ collective efficacy was also lower at Striving Readers schools at 
the beginning of the project but increased steadily and surpassed that of matched schools. At the 
end of the project, however, both personal and collective efficacy at the Striving Readers schools 
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decreased to the level of matched schools. It is important to note that the matched school 
participation decreased considerably in the final year. For the ALM coaches/LSC teachers, 
personal and collective efficacy increased with exposure to the Striving Readers program. After 
teachers’ first year in the program, personal efficacy dipped, but it recovered and increased across 
the remaining years of the project. Their collective efficacy increased their first year in the 
program, dipped in the second year, and recovered in the third and fourth years. 


Conclusions 


Over the course of the Danville Striving Readers project, literacy coaches, administrators, 
and content area teachers achieved high levels of participation in the professional development 
inputs overall for both the targeted intervention and whole-school model. However, there was 
more variation in the levels of classroom implementation of the targeted and whole-school 
interventions. For the targeted intervention, implementation was relatively low in year 1 but 
improved to high levels in years 2 through 4. For the school-wide model, middle school teachers’ 
implementation was higher than high school teachers’ classroom implementation. 


The Striving Readers project has yielded positive impacts on students. In particular, the 
targeted intervention (LSC) has improved the reading achievement of ninth-grade struggling 
readers and the reading strategy use of sixth-grade struggling readers. In addition, LSC has had a 
positive impact on both sixth- and ninth-grade students’ reading motivation. 


Introduction and Study Background 
Context for the Study 


The Danville, Kentucky Striving Readers project involved ten middle schools, nine high 
schools and two 6-12 schools in seven rural school districts. One of the 6-12 schools was an 
alternative school for students who have not succeeded in a traditional middle and high school 
setting. Student populations within these schools ranged in size from 26 to 1,222 with a mean of 
555 students per building. The school demographics are shown in Table 1.1. The school 
demographics remained fairly constant across years, with the possible exception of a decline in 
high school enrollment in year 3. 
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Table 1.1 


School Demographics 


Average 
Number of — enrollment African Free/ 
schools (min, max) White American reduced Disability 
Year 1 
Ms 12 460 91.9% 4.2% 50.4% 14.7% 
(56, 816) 
HS 11 682 92.7% 4.2% 37.3% 12.1% 
(56, 1144) 
Year 2 
Ms 12 457 91.5% 4.3% 49.6% 12.8% 
(26, 834) 
HS 11 703 92.7% 4.1% 34.4% 10.6% 
(68, 1222) 
Year 3 
Ms 12 475 90.5% 3.7% 53.2% 14.4% 
(41, 825) 
HS 11 577 91.5% 4.5% 40.3% 12.6% 
(49, 952) 
Year 4 
Ms 12 477 90.1% 4.8% 56.1% 13.8% 
(41, 849) 
HS 11 574 91.6% 4.4% 43.5% 12.1% 
(80, 939) 


Note. Middle schools are grades 6-8. High schools are grades 9-12. 
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Theoretical Rationale for and Description of the Intervention Models 


Targeted Intervention. The targeted intervention for the Danville project was the LSC, 
developed by the University of Kansas Center for Research on Learning (KU-CRL) as one 
component of the Strategic Instruction Model (SIM; Tralli, Colombo, Deshler, & Schumaker, 1996). 
The LSC was developed to assist adolescents with learning disabilities in the general education 
classroom and is divided into three strands: (a) Acquisition, (b) Storage, and (c) Expression. Each 
strand included a number of strategies designed to help students derive information from texts, 
identify and remember important information, or develop writing or academic competence. Each 
strategy was taught through eight instructional stages: pretest and commitments; describe; 
model; verbal practice; controlled practice; feedback; posttest and commitments; and 
generalization. 


The acquisition strand was geared toward helping students gain information from text. 
This strand included strategies such as Word Identification, Visual Imagery, Self-Questioning, and 
Paraphrasing, all of which were taught in this project. In general, previous research has examined 
the LSC strategies for acquiring information from text and has shown positive results when used 
for students with learning disabilities in grades 7 through 12 (Clark, Deshler, Schumaker, Alley, & 
Warner, 1984; Lee & Von Colln, 2003; Lenz & Hughes, 1990; Schumaker & Deshler, 1992; 
Woodruff, Schumaker, & Deshler, 2002). 


The strategies in the storage strand are designed to help students identify, organize, and 
store important information. The storage strand included the following strategies: FIRST-letter 
Mnemonic, Paired-Associates, and the LINCS Vocabulary Learning Strategy. Previous studies have 
suggested the effectiveness of the FIRST-Letter Mnemonic and Paired Associates strategies, which 
are designed to help students remember information (Bulgren, Hock, Schumaker, & Deshler, 1995; 
Nagel, 1982). In this project, the LINCS strategy was part of the targeted intervention. This 
strategy, which involved using a mnemonic to memorize word meanings, has yielded positive 
results in previous research studies (see Ellis, 1992). 


The Expression Strand included strategies for assisting students with writing and academic 
competence. It included the Sentence Writing Strategy, the Paragraph Writing Strategy, the Error 
Monitoring Strategy, the INSPECT strategy, the Theme Writing Strategy, the Assignment 
Completion Strategy, and the Test-Taking Strategy. Studies of individual Expression Strand 
strategies have demonstrated improved sentence writing (Kline, Schumaker, & Deshler, 1991), 
paragraph organization (Moran, Schumaker, & Vetter, 1981), revising and editing (McNaughton, 
Hughes, & Ofiesh, 1997), and theme writing (Hock, 1998). The targeted intervention in this 
project included Sentence Writing and Paragraph Writing. 


Theoretically, the pedagogical tenets underlying the LSC are grounded in notions related to 
self-regulated learning, generalization, and motivation. The primary goal is teaching students 
“how to learn” (Schumaker & Deshler, 1992; Schumaker & Deshler, 2006). This is accomplished by 
teaching students a variety of task-specific learning strategies that enable them to deal with the 
immediate demands of the school curriculum and the self-regulatory processes needed to 
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successfully transfer these skills to other contexts (Deshler & Schumaker, 1986; Schumaker & 
Deshler, 2006). Thus, students are taught metacognitive skills that enable them to monitor 
progress made toward achieving goals (Deshler, Warner, Schumaker, & Alley, 1983; Flavell, 1979) 
and that enable them to control their own learning and thinking (Baker & Brown, 1984; Deshler et 
al., 1983). Personal attributions are important to this process in that students must accept 
responsibility for their own learning (Deshler & Schumaker, 1986) and develop the intrinsic 
motivation necessary to transfer their knowledge of strategies and how to use them in new 
situations and settings (Garner, 1990; Schumaker & Deshler, 1992). 


In the Danville project, sixth- and ninth-grade students who scored two grade levels or 
more below grade level in reading received a minimum of 250 minutes per week of supplemental 
reading instruction in a targeted intervention class taught by an LSC teacher (who also served the 
school-wide model as an ALM coach). Students were placed in this course in addition to their 
regular reading/language arts classes for an entire school year. Table 1.2 shows the elements of 
the Learning Strategy Curriculum and the years these strategies were available to teachers for use 
in their classrooms. It is important to note that, due to scheduling necessities, some intervention 
classes met for longer than 250 minutes per week. In those classes, teachers were instructed to 
provide no more than 300 minutes of LSC instruction and to utilize the remaining time on other 
literacy activities. 


Table 1.2 


Learning Strategies Curriculum Elements by Implementation Year 


Implementation Year 
Learning Strategy Year 1 Year 2 Year 3 Year 4 


Word Identification 


Strategy (DISSECT) a ‘ ” * 
Visual Imagery Strategy 

(SCENE) A - , 
Self Questioning Strategy 

(ASKIT) : ‘ “ ‘ 
Paraphrasing Strategy (RAP) X X X X 
Vocabulary Strategy (LINCS) X X X X 
Sentence Writing Strategy 

(PENS and MARK) . * . ‘ 
Inference Strategy (INFER) X X xX 
Fundamentals of 

Paraphrasing and x x x 
Summarizing Strategy 

(TMtoD) 

Possible Selves (motivation x x 
strategy) 
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Over the course of the project, the professional development model for the targeted 
intervention included summer and follow-up trainings and on-site support from a mentor coach. 
To learn how to implement the targeted intervention, teachers participated in summer 
workshops, which were led by a certified LSC trainer from U of L. During each school year, the 
trainer led the teachers in follow-up workshops. Across the project, LSC teachers received 26 days 
of workshop training in the targeted intervention in total. To support their ongoing learning and 
development, teachers also participated in monthly coaching visits and regular distance support 
by mentor coaches from the Collaborative for Teaching and Learning (CTL) who were trained in 
the intervention alongside the teachers. Further, the LSC trainer made at least five visits to each 
teacher’s classroom over the course of the project. 


It is important to note that the teachers for the targeted intervention also served as 
literacy coaches who supported the implementation of the school-wide model in their schools. As 
part of their training for the project, most of the teachers participated in a literacy coach 
certification training offered through U of L. Through this facet of the project, they completed 
hallmark assessments designed to support their work as literacy coaches for the school-wide 
model as well as to further develop their skills as targeted intervention teachers. In addition, the 
intervention teachers fully participated in all training related to the school-wide model. 


Whole-School Intervention. The whole-school intervention used in the Danville Striving 
Readers project was the CTL ALM (Awbrey, 2008). The ALM is designed to support cross-content 
teachers in regularly applying literacy strategies during instruction in service of content learning. 
It is a professional development-based program, in that it does not include a specific set of texts 
for teachers to implement. The model focuses on the following components: vocabulary 
development, reading comprehension, verbal fluency, writing to learn, writing to demonstrate 
learning, and academic dialogue. 


All teachers in the Striving Readers schools were expected to integrate the ALM into their 
classroom instruction, across the curriculum, and across grade levels (6 through 12"), It was 
expected that all teachers, including auxiliary content teachers such as music and physical 
education, would use the ALM strategies to some extent to integrate literacy into the content 
areas. 


All content area teachers in the Striving Readers schools received 30 hours (5 days) of 
training related to administering the school-wide intervention during the summer of 2006. Initial 
training was provided by the Professional Development Director from CTL and five mentor 
coaches from CTL staff. Additional assistance in the form of ongoing, job-embedded professional 
development was provided throughout the year on a daily basis by ALM coaches in each building 
and the external mentor coaches from CTL. 


CTL provided professional development training, mentoring, coaching development, and 


direct coaching of teachers. The professional development activities to support the whole- 
school model included the following: 
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2 days of teacher training to full faculty in 21 project schools, with school- 
specific training agendas, based on observed needs of the school; 
direct-coaching training to school ALM coaches on coaching skills development, 
content area expertise, and support of the school-wide literacy model; 
department-specific training opportunities for each participating school, with 
CTL content specialists conducting each 3-hour training session; 

follow-up trainings as needed, and mentoring of ALM coaches in how to 
conduct formal sessions for specific departments in their schools; 

ongoing ALM coach mentoring, with several interactions each month (as 
individual school needs dictated), focused on modeling coaching and training, 
coconstructing coaching activities, observing, and providing feedback on 
coaching activities; 

direct training and development for school literacy leadership teams; 
facilitation in the development of schools’ long-range literacy plans, with 
follow-up support for monitoring the successful implementation of those plans; 
and 

direct training for school and district administrators to support development of 
leadership skills, strategies, and tool application. 


In addition to support provided by CTL, U of L provided direct training and coursework 
support for the reading intervention and foundations of literacy knowledge. 


The professional development model included training and support for administrators, as 
well. Each year, school administrators were invited to participate in the school-wide model 
training with their faculties. Additionally, each year administrators attended training days where 
they received professional development in literacy leadership to support both the targeted 
intervention and the school-wide model and participated in monthly on-site mentor meetings. 


Logic Models 


Exhibit 1: Logic Model for the Target Intervention (See Appendix A). 


Exhibit 2: Logic Model for the Whole-School Intervention (See Appendix B). 


Key Evaluation Design Features 


The evaluation is designed to measure the impact of the targeted intervention on student 
outcomes and the impact of the whole-school model on student outcomes and teacher efficacy. 
The impact study was guided by the following research questions: 


What is the impact of the LSC on the reading achievement, strategy use, and 
motivation of struggling adolescent readers? 


Cantrell, Almasi, Carter, & Rintamaa (2011) 


Striving Readers Final Evaluation Report: Danville, Kentucky 


e What changes occurred in the personal and collective efficacy of the LSC teachers/ALM 
coaches over the course of the project? 


e What is the impact of the ALM on student achievement in reading and the content 
areas? 


e What is the impact of the ALM on content teachers’ personal and collective efficacy? 

For the evaluation of the targeted intervention, data from cohorts of sixth and ninth 
graders were combined for analysis each year of 4 years. Every year, sixth- and ninth-grade 
students were randomly selected for treatment and control groups. Student outcome measures 
collected in the fall and spring of each year are as follows: 

e Group Reading Assessment and Diagnostic Evaluation (GRADE) 

e Metacognitive Awareness of Reading Strategies Inventory (MARSI) 

e Motivation to Read Questionnaire (MRQ) 

The evaluation of the whole-school model involved yearly measurement of student 
achievement in reading and writing content areas in Striving Readers schools as compared to 
matched schools. It also included teachers’ efficacy for literacy teaching at Striving Readers 
schools compared to matched schools for the 4-year study period. The Kentucky Core Content 
Test (KCCT) was used as a student achievement measure to determine the impact of the whole- 


school model on students’ reading and writing achievement. A Teacher Efficacy Survey was used 
as a teacher outcome measure. 


Evaluation of the Implementation of the Targeted Intervention 


Summary of the Design of the Implementation Study 


The research questions that guided the implementation study of the targeted intervention 
are: 


e What was the level of implementation of professional development/support for 
teachers/coaches/leaders? 


e What was the level of implementation of classroom instruction? 


Exhibit 3 indicates the data sources that were used to answer each implementation research 
question. 
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Exhibit 3. 


Data Sources on Implementation: Targeted Intervention (Learning Strategies Curriculum) 


Measures/Data Sources 


Responsible Partner Record Review 


Data Sources 


Developer of 


ALM model 


(CTL) 


UofL 
trainer/LSC 


certified 


trainer 


Project 
Director 


Evaluator 


PD attendance 


records 


Surveys/ 
Questionnaires/ 


Logs/Classroom 


Observations 


Type and amount of 
professional 
development provided to e 6 e e 
LSC teachers and level of 
participation. 


Type and amount of 
coaching provided to LSC 
teachers and level of 
participation. 


Level of participation of 
administrators in 
professional 
development. 


Percentage of time that 
LSC teachers used the 
targeted instructional 
strategies with fidelity. 


Proportion of school days 
students received 
intervention instruction 
by a trained LSC teacher. 


LSC Targeted Intervention Implementation Results 


Professional Development Inputs. Table 1.3 shows the average participation of LSC 
teachers in professional development sessions and in coaching visits. As well, Table 1.3 shows the 
average number of days that school administrators attended training. This table indicates high 
levels of participation by both teachers and administrators in that all teachers and most 
administrators participated fully in the training each year of the project. If teachers missed any of 
the training due to illness, school responsibilities, or other factors, the LSC trainer and mentor 
coaches met individually with the teacher to provide missed training content. 
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In addition to attending training sessions on the targeted intervention, LSC teachers 
received on-site mentoring by a CTL mentor coach. Table 1.3 shows the average number of days 
LSC teachers received on-site mentoring. As the table shows, on-site mentoring of the targeted 
intervention decreased for all teachers across the years, as teachers were expected to need less 
and less implementation support as the project progressed. 
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Means for Professional Development Inputs for Targeted Intervention by School 


PD Inputs 
Mentor 
LSC Teacher Training Administrator Training Coaching 
Mean Days Mean Days Mean Site 
School Attended % at Full Attended % at Full Visit Days 
Level (Min, Max) n° Participation® (Min, Max) n Participation’ (Min,Max) 
Year 1 
Ms? 6.6 of 7 10 100 1.7 of 2 10 90 5.0 
(6.5, 7) (1, 2) (4.2, 6.5) 
HS” 6.6 of 7 12 100 1.7 of 2 9 77.8 6.5 
(6.5, 7) (1, 2) (4.5, 10) 
6-12° 7.0 of 7 2 100 1.8 of 2 2 100 5.6 
(1.5, 2) (5.3, 5.8) 
Year 2 
MS? 7.0 of 7.5 10 100 1.3 of 1.5 10 100 3.8 
(6.5, 7.5) (1, 1.5) (1, 6) 
HS? 7.1 of 7.5 12 100 1.3 of 1.5 9 100 4.6 
(5.5, 7.5) (1, 1.5) (2, 8) 
6-12° 7.5,7.5 2 100 1.5 of 1.5 2 100 4.5 
(4,5) 
Year 3 
MS? 6.4 of 7 10 100 1.3 of 1.5 10 100 2.5 
(6, 7) (1, 1.5) (.5, 12) 
HS® 6.6 of 7 12 100 1.3 of 1.5 9 100 2.3 
(4.5, 7) (1, 1.5) (.5, 4) 
6-12° 6.5 of 7 2 100 1.5 0f1.5 2 100 1.5 
(6, 7) (1, 2) 
Year 4 
MS? 4.1 of 4.5 10 100 1.4 of 1.5 10 100 1.3 
(2.5, 4.5) (.5, 1.5) (0, 4) 
HS® 3.3 of 4.5 12 100 1.4 of 1.5 9 100 2.3 
(1, 4.5) (1, 1.5) (.5, 10) 
6-12° 4 of 4.5 2 100 1.5 of 1.5 2 100 5 
(3.5, 4.5) (0, 1) 


ae ee 
*n = 10 middle schools; °n = 9 high schools; ‘n = 2 Grades 6 — 12 schools. 


Every school employed one Literacy Coach with the 


exception of three high schools, which had two Literacy Coaches each. “LSC teachers who missed group training sessions 
received make up training for all sessions from the LSC trainer and mentor coaches. ‘Participation for year 1 was complete with 


21.5 days attended, and for years 2, 3, and 4 with > 1 day attended. 
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It is important to note that LSC teachers received support from mentor coaches through 
telephone calls, email correspondence, and formal distance support. This support was for both 
the targeted intervention and whole-school intervention. Although documentation of this support 
was provided in years 2 and 3, the extent to which the support was specific to the targeted 
intervention or whole-school model was not designated. LSC teachers and mentor coaches 
engaged in 10,461 minutes of phone support calls, exchanged 2,410 emails, and participated 357 
times in formal distance support during the second year of the project. In year 3, LSC teachers and 
mentor coaches engaged in 11,490 minutes of phone support calls, exchanged 4,320 emails, and 
participated 45 times in formal distance support. In year 4, LSC teachers and mentor coaches 
engaged in 433 phone support calls, exchanged 2,791 emails, and participated 57 times in formal 
distance support. 


The LSC trainer also provided support through site visits, phone calls, and electronic 
communication. In year 1, the LSC trainer reported that coaches received an average of 3 days of 
site visits. For year 2, continuing coaches received an average of 3 days of visits, and new coaches 
received an average of 4 days of site visits. In year 3, continuing coaches received an average of 
2.5 days of site visits, and new coaches again received an average of 4 days. In year 4, all coaches 
received an average of 2 days of site visits, according to the LSC trainer. 


Many of the LSC teachers/ALM coaches participated in a literacy leadership certification 
program offered by U of L. The participants completed hallmark assignments as part of their 
certification program. By 2010, 11 teachers had finished the certification program and graduated. 
Two more teachers are on track to complete their certification and degree by the end of 2011. 


Process for Measuring Classroom Implementation. Each LSC teacher was observed two 
times each year during the 4 years of the program. During the spring of 2007, all LSC teachers 
were observed for at least one class period on two occasions. During the 2007-2008, 2008-2009, 
and 2009-2010 academic years, LSC teachers were again observed twice each year, once in the fall 
and again in the spring. The purpose of these observations was twofold: (a) to determine 
treatment fidelity, and (b) to determine the extent to which LSC teachers implemented aspects of 
the LSC in their instruction. Observers sought and recorded evidence from the classroom 
environment, the observation, and an interview with the LSC teacher. 


In the fall of 2006, research assistants attended a training session conducted by the LSC 
intervention trainer. Following that training, the research assistants and investigators worked 
collaboratively to create an observation protocol that included the eight instructional stages of the 
LSC intervention (pretest and commitments, describe, model, verbal practice, controlled practice 
and feedback, posttest and commitments, and generalization). They then identified activities 
associated with each component and constructed interview questions that would clarify the 
observations. The evaluators shared the observation protocol with the LSC trainer and requested 
feedback on the instrument. The trainer made no suggestions for changes in the protocol. 


The investigators then met with the research assistants on two occasions to provide 
training related to taking field notes. Training consisted of lecture related to taking field notes, 
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watching video segments, practice taking field notes (both in classrooms and through videotape), 
and critique. Sample field notes were shared, critiqued, and refined. The research assistants were 
trained to organize their field notes using 5-minute time intervals to capture the nature of 
instruction throughout the entire lesson. Four codes were developed to characterize the range of 
instructional behaviors observed in the intervention classrooms. Those codes included: (a) LSC, (b) 
Other Literacy Activities, (c) Non-Literacy Activities, and (d) Behavior Management. 


Four members of the evaluation team used the field notes to identify the number of 
minutes spent engaged in the LSC, Other Literacy Activities, Non-Literacy Activities, and Behavior 
Management. These four researchers sought reliability for coding the observation protocols using 
10.5% of the data (n = 4 observations) during year 1. Inter-rater reliability was 89.8% among all 
four coders. Two of the coders then coded all remaining intervention observations. Overall, a 
total of 2,414 minutes of intervention instruction was observed and coded. In year 2, the same 
four team members achieved 92% percent agreement using 5 of 46 observations (10.9% of the 
data). Two coders then coded the remainder of the year 2 observations. For the purposes of this 
report, only the number of minutes teachers and students were engaged in LSC are reported. 
Time spent in Other Literacy Activities, Non-Literacy Activities, and Behavior Management is not 
disaggregated for this report. 


Classroom Implementation Results. Table 1.4 shows the mean percentage of time that LSC 
teachers spent on the targeted intervention in the Striving Readers classrooms. In year 1 of the 
project, fidelity to the LSC model was higher for ninth-grade LSC teachers than for sixth-grade LSC 
teachers. In years 2 and 3, implementation increased for both sixth- and ninth-grade LSC teachers. 
In year 4, implementation for sixth-grade teachers continued to increase, but implementation for 
ninth-grade teachers decreased slightly. This represents a large increase in implementation 
fidelity for middle school teachers particularly. 


Another important aspect of implementation fidelity involves the number of days that 
students received instruction in the targeted intervention by a trained LSC teacher. As a measure 
of this variable, the LSC teachers’ attendance in the intervention class is also presented in Table 
1.4. Intervention teachers were out of the targeted intervention classroom for a large percentage 
of days. LSC teachers missed, on average, 1 of every 6 days, which could affect the effectiveness 
of the targeted intervention. 
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Table 1.4 


Implementation of the LSC Model in Intervention Classrooms as Measured by Percent of Class Time 


Year 1 Year 2 Year 3 Year 4 
6m grade 
LSC Instruction 58.5% 80.0% 87.2% 87.7% 
LSC Teacher Attendance 90.5% 78.1% 82.2% 83.6% 
Number of LSC 
Teachers Observed 11 12 11 11 
9"" grade 
LSC Instruction 70.4% 78.5% 86.6% 82.9% 
LSC Teacher Attendance 91.1% 87.5% 87.4% 85.2% 
Number of LSC 
Teachers Observed 13 12 9 10 


Note. In year 1, every LSC teacher was observed twice in the spring, with the exception of one ninth-grade teacher who was 
observed once. In years 2 and 3, every LSC teacher was observed once in the fall and once in the spring. In year 4, every LSC 
teacher was observed once in the fall and once in the spring except for one sixth-grade teacher, who was only observed in the 
spring. The number of LSC teachers differs from year to year due to turnover. 


Literacy Coach Turnover. At the beginning of the study in 2006, 24 literacy coaches were 
hired at the 21 Striving Reader’s school (Table 1.5). Twelve of the literacy coaches remained in the 
Striving Readers program for the duration of the project. Nine of the literacy coach positions 
turned over once. One position turned over twice, and two positions turned over three times. A 
total of 17 literacy coaches left the 24 positions during the study, for a turnover rate of 70.1%. 
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Table 1.5 


Literacy Coach Turnover by School and Year 


cena 2006-2007 2007-2008 2008-2009 2009-2010 
Summer Midyear Summer Midyear Summer Midyear Summer Midyear 
Middle Schools 
MSA 1G 
MSB sua te 2° a" Lc atic 
MSC slag fe am 1¢ 3"1¢C 
MS D 1G 
MSE 1" ke re 
MS F 1" LC a4 1¢ 
MSG ike oe 
MSH 1 iC 
MS | 1c 
MS J Pe 
High Schools 
HS A (LC 1) sag 6 
HS A (LC 2) 1* Le 
HS B (LC 1) 1° LC ote 
HS B (LC 2) 1c 
HS C i fagl 
HS D 1 LC 2nd LC 
HS E (LC 1) 1 Lc 2c 3" Ic AM ice 
HS E (LC 2) ue 4 1¢ 
HS F 4 LC and LC 
HS G 1" LC 
HSH 1" LC oC 
HS | 1" LC 241¢ 
Grades 6-12 
Sch A 1" LC 
Sch B 1 LC 


Note. *Position was filled by coach currently at HS E (LC 2) 


Implications for Impact Analysis. \n effectiveness studies, it is essential to examine the 
extent to which the intervention was implemented with fidelity so that appropriate conclusions 
may be drawn from the research findings (Durlak & DuPre, 2008; Institute of Education Sciences, 
2009). If student outcomes do not improve after participation in an intervention, one of two 
explanations may be attributed: (a) the intervention, as designed, is not effective, or (b) the 
intervention was not implemented as it was designed or as the developer intended. For the 
targeted intervention in this study, implementation of the professional development was high, 
and classroom model was relatively high with increasing fidelity from year to year. There was 
considerable teacher turnover in some schools, but this did not appear to negatively influence 
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classroom implementation of the LSC model. Thus, it appears that outcomes from the impact 
analysis may be attributed to the effectiveness of the targeted intervention in this study. 


Evaluation of the Impacts of the Targeted Intervention 
Study Design 


Sampling Plan. The sampling plan for this evaluation was a two-stage sampling design 
wherein a purposively selected sample of 21 schools was selected in stage 1. Stage sampling 
occurred each year of the study. In stage 2, all sixth- and ninth-grade students who met eligibility 
criteria of scoring two grade levels below grade level (with the exception of students placed in 
self-contained special education classrooms full time) were randomly assigned to treatment and 
control groups within each of the 21 schools. Students could opt out of the intervention only with 
a written request by the parent or guardian. School administration strongly encouraged 
treatment for qualified students, however. 


The evaluation combined cohorts of sixth and ninth graders from multiple years. At the 
beginning of each academic year, a cohort of sixth-grade students and a cohort of ninth-grade 
students were randomized to treatment and control groups. In this evaluation, the four cohorts of 
sixth-grade students were combined for analysis of impacts on sixth-grade students, and the four 
cohorts of ninth-grade students were combined for analysis of impacts on ninth-grade students. 


Sample Size and Power. The empirical minimal detectible effects were derived at the 
completion of the study using Optimal Design Software developed by Spybrook, Raudenbush, 
Congdon, and Martinez (Optimal Design Software). The specific design used was person 
randomized at multisite trials. The minimal detectable effects calculated for sixth and ninth 
grades for this study were 0.13 and 0.11, respectively, with a power of .80. 


Sample Selection Process. Every sixth- and ninth-grade student in the Striving Readers 
schools completed the GRADE at the beginning of the fall semester, with the exception of students 
placed in self-contained special education classrooms full time. Every student with a Normal 
Curve Equivalent (NCE) of 33 or lower was assigned to the intervention or control group. Within 
each school, a stratified random sampling procedure was implemented using four demographic 
variables: special education status, free/reduced lunch status, ethnicity, and gender. The students 
were systematically assigned to the intervention or control group by sorting the students by 
demographic group and GRADE score within each subgroup. A random number generator was 
used to assign the first student into either the intervention or control group. Each subsequent 
student was alternately assigned to intervention or control. 


Counterfactual. Students who were selected for the control group received a regular 
elective as part of their sixth- or ninth-grade program. A wide range of electives were taken, 
including band, chorus, civics, and physical education. In general, it was not expected that the 
electives included sufficient literacy content to influence the literacy achievement of students in 
the control group. Reading intervention teachers did not interact with or teach students in the 
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control group, and intervention teachers did not share teaching or learning strategies with other 
teachers who may have influenced the performance of students in the control group. 


Missing Data. Case-wise deletion was used for missing data, with the exception of the 
state reading KCCT in the base year (2006). Two schools did not have data that year, so estimates 
from other years were averaged and substituted. 


Data Collection Plan 


In the 21 Striving Readers schools, all current sixth- and ninth-grade students (with the 
exception of students who were placed in special education classes all day) were administered the 
following measures in the fall and spring of each year: 


Group Reading Assessment and Diagnostic Evaluation (GRADE). Each year, all students 
took the GRADE assessment (Form A) during the first 2 weeks of the school year. Consistent with 
GRADE norming procedures, the GRADE was administered in classrooms by teachers. Prior to 
September 1 of the school year, schools administered make-up tests to any sixth- or ninth-grade 
student who missed the first administration. In the spring, students took the GRADE assessment 
(Form B) during the first 2 weeks of May, except in year 3, when students took the GRADE during 
the last 2 weeks of April. In both the fall and the spring, school literacy coaches gathered 
students’ GRADE answer sheets and mailed or delivered them to the Collaborative Center for 
Literacy Development offices. Research assistants scanned the answer sheets for scoring. 


Student Survey—Metacognitive Awareness of Reading Strategies Inventory 
(MARSI)/Motivation to Read Questionnaire (MRQ). The MARSI (Mokhtari & Reichard, 2002) is a 
self-report measure designed specifically to assess middle and high school students’ perceived use 
of reading strategies during academic reading. The MARSI includes items related to three strategy 
domains: global, problem-solving, and support strategies. The MRQ (Wigfield & Guthrie, 1997) is 
designed to measure four aspects of motivation for reading (a) self-efficacy (i.e., reading efficacy, 
reading challenge), (b) intrinsic motivation (i.e., reading curiosity, reading involvement, 
importance of reading, and reading work avoidance), (c) extrinsic motivation (i.e., competition in 
reading, recognition for reading, and reading for grades), and (d) social motivation in reading (i.e., 
social reasons for reading, compliance). Because the MRQ is designed for students in grades 3 
through 6, the MRQ was field tested with high school students and was modified. 


Each year, students completed the MARSI/MRQ? during the fall and spring of the school 
year. Research assistants administered and collected the student surveys during the weeks of 
September 1 through October 30 in fall and during the weeks of May 14 through June 1 in spring. 


"In year 1, we observed a large proportion of students who did not complete the student questionnaire 
(approximately 25%), possibly due to the length of the survey (82 items). An Item Response model indicated that the 
questionnaire could be divided without excessive loss of precision. In the fall of year 2, we randomly divided the 
items into two survey forms of 40 items, and our completion response was higher. In the spring of year 2 and at both 
administrations in years 3 and 4, we included all of the MARSI items and half of the MRQ items for a total of 60 items. 
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Because the sample of interest in this evaluation was struggling readers, the research assistants 
read the survey aloud to students as it was administered. 


Summary of Analytic Approach 


HLMs were used to estimate the impact of the LSC on student achievement, motivation, 
and reading strategies outcomes. The GRADE NCEs were used to estimate the impact of the LSC 
intervention on achievement. The average MARSI scores were used to estimate the impact on 
reading strategy use, and the MRQ averages were used to estimate the impact on motivation. 


A two-level HLM model (students assigned to intervention or control group within schools) 
was used to determine the impact of the targeted intervention. At the student level, the spring 
outcome variable (achievement, strategy use, or motivation) was modeled as a function of fall 
outcome variables, intervention/control status and four demographic variables: gender, ethnicity, 
free/reduced lunch status, and special education. 


Level-1 Model: Student Outcomes (achievement, reading strategies, or motivation) 


M 
Yi= Bo + Ba (¥*y) + B 2 (Ti) + Y Bima mij + € ij 
m=3 


where 


Y;, is the spring student outcome (posttest) score for student i at school j; 

P ois the mean student outcome (posttest) score for control students at school j; 

Y*;,is the fall student outcome (pretest) score for student i centered at school j; 

£ iis the average student outcome (pretest) slope for students at school j; 

Tj = 1 if student i is assigned to LSC intervention at school j, and 0 if control; 

£ 2, is the mean difference of student outcome pre-post gain between intervention and 
control students at school j; 

Q mi are additional covariates representing demographic characteristics of student i at school 
j (gender, ethnicity, free/reduced lunch, and special education status); 

2 mare coefficients corresponding to student demographic covariates (gender, ethnicity, 
free/reduced lunch, special education status), and 

& ; is the random effect representing the difference between student ij’s score and the 


predicted mean score for school j. These residual effects are assumed normally distributed with 
mean 0 and variance o”. 


Level-2 Model: Student Achievement — School Level 
This analysis was performed on data from sixth-grade students and ninth-grade students 
collected over multiple years. The covariates in this model pertain to the concurrent year the 


student was in the intervention or control group with the exception of the Reading KCCT score, for 
which the score for the base year, spring 2006, was used. In addition to the base year Reading 
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KCCT score, other school level covariates included enrollment, percent of white students in the 
school, percent of African American students, percent of students qualifying for free or reduced 
lunch fees and percent of students with disabilities, averaged over the 4 years of the study. 


Q 
Bo= Yoot D7 ogWaji + H oj 


q 


Bij= 710 
B= ¥20 
B mj = ¥ mo 
where 


Yoo is the mean student outcome (posttest) score of sixth-grade control students in Kentucky 
Striving Readers middle schools (note: or ninth grade in high schools); 

W,j are school level covariates including base year Reading KCCT (spring 2006), and average 
school percent free/reduced lunch, percent white students, percent black students, and percent 
disability; 

Y oq are Coefficients corresponding to school-level covariates; 

Ho is the unique effect of school j on mean student outcome, holding W,; constant (or 
conditioning on W,;) - this effect is assumed normally distributed with mean 0 and variance 7’; 

¥ 10 is the fall student outcome (pretest) slope; 

¥20 is the overall target intervention treatment effect on spring student outcome (posttest) 
scores; 

¥ mo is the fixed m" student covariate effect (gender, ethnicity, free/reduced lunch, special 
education status) on the spring outcome variable. 


Selection of Covariates. The random assignment procedure included all student 
demographic variables in the HLM model, so were included regardless of significance. Interaction 
effects were not considered. 


Description of the Targeted Intervention Samples, by Year 


Characteristics of Literacy Coaches (LSC Teachers/ALM Coaches) 


The literacy coaches implemented the LSC targeted and ALM whole-school interventions 
within the 21 schools. As shown in Table 1.6, almost all coaches were white females, and most 
were not certified in reading. Additionally, about half had a master’s degree, and about one third 
had Rank | (30 hours above master’s degree). Literacy coaches had an average of 12.9 years of 
experience. 
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Table 1.6 


Literacy Coach Demographics 


No. of Sex Ethnicity Reading Certification 
School Literacy Asian Not 
Level Coaches®*”°*° Male Female White American Certified Certified 

Year 1 

MSs 11 0 11 11 0 5 6 

HS 12 1 11 12 0 1 11 

6-12 Z 0 2 2 0 0 vs 

Total 25 1 24 25 0 6 19 
Year 2 

Ms 11 1 10 11 0 4 Fi 

HS 13 2 11 12 1 1 12 

6-12 Z 0 2 2 0 0 2 

Total 26 3 23 25 1 5 21 
Year 3 

Ms 10 1 9 10 0 3 7 

HS 12 3 9 11 1 1 11 

6-12 2 0 2 2 0 0 2 

Total 24 4 20 23 1 4 20 
Year 4 

Ms 12 2 10 12 0 3 9 

HS 13 2 11 12 1 1 12 

6-12 2 0 2 2 0 0 2 

Total 27 4 23 26 1 4 23 


Note. “Literacy Coaches that stayed in the position for less than half a semester are not included in these statistics. "Twenty-four 
Literacy coaches were hired at the beginning of year 1, and one middle school teacher was replaced midyear. “Four new Literacy 
Coaches were replaced in the beginning of year 2 (one middle school and three high school coaches), and two were replaced 
midyear (one middle school and one high school coach). “Four new Literacy Coaches were replaced in the beginning of year 2 (two 
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middle school and two high school coaches), and none were replaced midyear. “Two new Literacy Coaches were replaced in the 
beginning of year 2 (one middle school and one high school coach), and three were replaced midyear (one middle school and two 
high school coaches). 


Characteristics of LSC Classrooms 


All students were expected to receive a minimum of 50 minutes of intervention instruction 
daily, or 250 minutes per week. Classes ranged from 45 to 90 minutes in length. For those classes 
less than 50 minutes an additional class period was added so students averaged at least 250 
minutes of intervention instruction weekly. Classes longer than 60 minutes were instructed to use 
time beyond 60 minutes for other literacy activities. Middle school classes ranged from 50 to 90 
minutes daily (250 to 450 minutes weekly). High school classes ranged from 50 to 84 minutes 
daily (250 to 420 minutes weekly). 


Characteristics of Students 


Table 1.7 shows the number of eligible students assigned to the intervention and control 
by year and by grade. The number of students with outcome data is shown, including students 
whose parents opted out of the program. Aside from those students who opted out of the 
program, all students who were randomly assigned to the treatment group were successfully 
placed in treatment classes. However, as depicted by the table, some students are not included in 
the analysis because they did not take the spring test or they transferred or withdrew from the 
school. 
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Table 1.7 


Intended and Actual Number of Students in LSC Intervention and Control Groups 


Intervention Control 
Treatment condition Yri Yr2 Yr3 £=Yr4_ Total Yr1 Yr2 Yr3  Yr4_ Total 
Sixth grade 
Actual numberof = 4542 asqb 137° 143% 605° 131 138 120 141 530 
students in study 
Students without 
spring GRADE 6 4 3 8 71 9 14 6 11 40 
scores 
Students that 
transferred or 15 21 12 9 58 26 22 15 7 70 
withdrew 
Total number of 
: 192 179 152 160 683 166 174 141 #159 640 
intended students 
Ninth grade 
Actual number of f g h j 
. 192) 183° 141 77 593 165 150 144 76 535 
students in study 
Students without 
spring GRADE 16 10 15 6 47 33 25 17 24 99 
scores 
Students that 
transferred or 42 19 28 15 104 48 27 19 6 100 
withdrew 
Total number of k k 
250 212 184 98 744 246 202 180 106 734 


intended students 


*Eleven parent opt-outs. Four parent opt-outs. “Three parent opt-outs. “Eight parent opt-outs. “Twenty-six 6" grade 


parent opt-outs. 
‘Nineteen parent opt-outs. *Eleven parent opt-outs. "Fourteen parent opt-outs. ‘Ten parent opt-outs. ‘Fifty-four 9" 


grade parent opt-outs. 

‘in year 4, ninth-grade students that attended Striving Readers middle schools could have taken the intervention class 
in sixth or seventh grade. These students were not eligible to be in the evaluation in year 4, decreasing the number of 
eligible high school students. 


The participation rate for sixth grade for the LSC intervention in the 4-year study was 530 
of the 640 intended students, or 82.8%. The intervention participation rate for ninth grade was 
535 of the 734 intended students, or 72.9%. There were no cross over students; no control 
student was placed in an intervention class. Also, no LSC teacher had an opportunity to teach the 
LSC curriculum to a control student. 


Demographics of students in the intervention and control conditions with outcome data 
were similar for both sixth and ninth grades in terms of gender, ethnicity, and socioeconomic 
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status (SES; Table 1.8). The sample consisted of more males than females. In terms of ethnicity, 
most students in the sample were white, and over half of the sample received free/reduced lunch, 
an indicator of low SES. Slightly more intervention students were in special education than control 


students. 


Table 1.8 


Years 1, 2, 3, and 4 Intervention and Control Student Demographics (and Proportions) 


Gender Ethnicity Lunch Special Education 
Group Male Female White Minor- Reg Free/ NotIn  Read/ Other 
ity Pay Red Write 
6" grade 
Interv. 360 245 526 79 187 418 404 126 75 
(.60) (.40) (.87) (.13) (.31) (.69) (.67) (.21) (.12) 
Control 309 221 456 74 172 358 379 96 55 
(.58) (.42) (.86) (.14) (.32) (.68) (.72) (.18) (.10) 
Total 669 466 982 513 359 776 783 222 130 
(.59) (.41) (.87) (.13) (.32) (.68) (.69) (.20) (0.11) 
9"" grade 
348 245 524 69 222 371 417 110 66 
Interv. (.59) (.41) (.88) (12) (.37)  (.63) (.70) (.19) (.11) 
Control 295 240 464 71 208 327 411 73 51 
(.55) (.45) (.87) (.13) (.39) (.61) (.77) (.14) (.10) 
Total 643 485 988 140 430 698 828 183 117 
(.57) (.43) (.88) (12) (.38) (.62) (.73) (.16) (.10) 


Pearson’s Chi-Square was used to test the hypothesis that students in intervention and 
control groups were similar for each demographic group, and no group was significantly different 
at the .05 level for the sixth grade. The test of equality of the proportion of boys and girls in the 
intervention versus control group yields X? (1, n = 1135) = 0.169, with p = .717. The test of 
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equality of the ethnic group representation (proportion of whites and minorities) in the 
intervention versus control group yields X? (1, n = 1135) = 0.198, with p = .664. The test of 
equality of SES group representation (proportion of students qualifying for the free/reduced lunch 
program) in the intervention versus control group yields X2 (1, n = 1135) = 0.311, with p = .609. 
Finally, the test of equality of special education designation (proportion of students qualifying for 
special education for reading/writing or for another designation) in the intervention versus control 
group yields X? (2, n = 1135) = 2.986, with p = .225. 


For ninth grade, the test of equality of the proportion of boys and girls in the intervention 
versus control group yields X2 (1, n = 1128) = 1.442, with p = .253. The test of equality of the 
ethnic group representation (proportion of whites and minorities) in the intervention versus 
control group yields X? (1, n = 1128) = 0.198, with p = .664. The test of equality of SES group 
representation (proportion of students qualifying for the free/reduced lunch program) in the 
intervention versus control group yields X? (1, n = 1128) = 0.311, with p = .609. Finally, the test of 
equality of special education designation in the intervention versus control group yields X? (2, n= 
1128) = 6.482, with p = .039, where the proportion of students not in special education in the 
intervention is .70 versus .77 in the control group. 


Tests of Equivalence of Treatment and Comparison Students 


The null hypothesis that sixth-grade intervention students’ pretest scores did not differ 
significantly from control students’ pretest scores on the GRADE was evaluated (Table 1.9). 
Results showed that sixth graders in the intervention and control conditions were equivalent at 
the time of the pretest in terms of their reported strategy use. The null hypothesis that ninth- 
grade intervention students’ pretest scores did not differ significantly from control students’ 
pretest scores on the GRADE was also evaluated. Results showed that ninth graders in the 
intervention and control conditions did not differ significantly at the time of the pretest. 
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Table 1.9 
Equivalence of Groups on GRADE Pretest 


95% Cl 
Condition n M SD t df p Lower Upper 


6" grade 


Intervention 605 21.11 9.49 0.990 1133 0.464 -0.544 1.653 
Control 530 21.66 9.32 


9"" grade 


Intervention 593 23.59 8.05 -0.074 1126 0.180 -0.994 0.922 
Control 535 23.56 8.34 


Note. Cl = confidence interval. 


The null hypothesis that sixth-grade intervention students’ pretest scores did not differ 
significantly from control students’ pretest scores on the MARSI was evaluated (Table 1.10). 
Results showed that sixth graders in the intervention and control conditions were equivalent at 
the time of the pretest in terms of their reported strategy use. The null hypothesis that ninth- 
grade intervention students’ pretest scores did not differ significantly from control students’ 
pretest scores on the MARSI was also evaluated. Results showed that ninth graders in the 
intervention and control conditions did not differ significantly at the time of the pretest. 
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Table 1.10 
Equivalence of Groups on MARS! Pretest 


95% Cl 
Condition n M SD t df p Lower Upper 


6" grade 


Intervention 439 2.97 0.65 1.711 827 0.803 -0.012 0.169 
Control 390 3.05 0.66 


9"" grade 


Intervention 368 2.65 0.71 -0.882 708 0.175 -0.147 0.056 
Control 342 2.60 0.66 


Note. Cl = confidence interval. There are fewer students that completed the MARSI than completed the GRADE because MARSI 
scores are only included if the student also completed the GRADE. Also, the MARSI was administered on different days, by 
different people in a different context than the GRADE. 


The null hypothesis that sixth-grade intervention students’ pretest scores did not differ 
significantly from control students’ pretest scores on the MRQ was evaluated (Table 1.11). Results 
showed that sixth graders in the intervention and control conditions were equivalent at the time 
of the pretest in terms of their reported strategy use. The null hypothesis that ninth-grade 
intervention students’ pretest scores did not differ significantly from control students’ pretest 
scores on the MRQ was also evaluated. Results showed that ninth graders in the intervention and 
control conditions did not differ significantly at the time of the pretest. 
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Equivalence of Groups on MRQ Pretest Years 1 through 4 


95% Cl 
Condition n M SD t df p Lower Upper 
6" grade 
Intervention 439 2.78 0.45 0.183 825 0.903 -0.056 0.068 
Control 388 2.79 0.46 
gin grade 
Intervention 366 2.45 0.48 -0.527 706 0.463 -0.086 0.050 
Control 342 2.44 0.44 


Note. Cl = confidence interval. There are fewer students that completed the MARSI than completed the GRADE because MARSI 
scores are only included if the student also completed the GRADE. Also, the MARSI was administered on different days, by 
different people in a different context than the GRADE. 


Measures of Student Outcomes 


Impacts on Students 


Group Reading Assessment and Diagnostic Evaluation (GRADE). The GRADE is a norm- 
referenced, standardized test of reading achievement which yields standard NCE scores, 
normalized standard scores with a mean of 50 and a standard deviation of 21.06. NCEs range 
from 1-99. The GRADE components and subtests for sixth and ninth grades include vocabulary, 
sentence comprehension, passage comprehension, and listening comprehension (Williams, 2001). 
Word-level skills are not measured on the GRADE. Fugate and Waterman (2004) found the 
GRADE’s reliability adequate for educational decision making. Reliability coefficients across test 
levels, test forms, and subject grade levels are consistently .90 or better for the total test score, 
including subtests of vocabulary, sentence comprehension, and passage comprehension. 
Alternate forms reliability ranged from 0.81 to 0.93, and test-retest reliability coefficients ranged 
from 0.88 to 0.93. The GRADE technical manual (Williams, 2000) shows the NCE distribution to be 
identical to the standard scores distribution. Further, the manual states that, although NCEs are 
based on percentiles, they have been converted to an equal-interval scale, making arithmetical 


manipulation appropriate. 


Fugate and Waterman (2004) found the GRADE’s reliability adequate for educational 
decision making. Internal reliability coefficient alphas and split-half reliabilities were consistently 
high (alphas above 0.90) across test levels, forms, and grade-enrollment group. Alternate forms 
reliability ranged from 0.81 to 0.93, and test-retest reliability coefficients ranged from 0.88 to 0.93 
indicating stability of GRADE total test scores. The reliability of the GRADE as presented by the 
GRADE Technical Manual (Chapter 4) indicates consistency in test scores. 
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Metacognitive Awareness of Reading Strategies Inventory (MARSI). The MARS is a self- 
report measure designed specifically to assess middle and high school students’ perceived use of 
reading strategies during academic reading (Mokhtari & Reichard, 2002). The MARSI includes 
items related to three strategy domains: Global, Problem-Solving, and Support Strategies. Global 
Reading Strategies represent a set of reading strategies oriented toward a global analysis of text. 
Problem-Solving Strategies include items oriented around strategies for solving problems when 
the text becomes difficult to read. Support Reading Strategies involve use of outside reference 
materials, taking notes, and other functional or support strategies. The survey items are 
presented on a scale of 1 to 5, where 1 is equal to “I never or almost never do this” and 5 is equal 
to “| always or almost always do this.” This measure has been reported to have high reliability. 
Mokhtari and Reichard reported a Cronbach’s alpha coefficient of 0.93 for the entire scale. In the 
first year of this study, we found a Cronbach’s alpha coefficient for fall and spring of 0.92 and 0.93 
(930 items), respectively, for the MARSI. 


Motivation to Read Questionnaire (WRQ). A modified MRQ (Wigfield & Guthrie, 1997) was 
used to measure motivation for reading. Items relate to aspects of motivation such as self- 
efficacy, intrinsic motivation, extrinsic motivation, and social motivation in reading. The MRQ 
consists of 55 items and uses a 4-point Likert response scale. The MRQ was normed for students 
through grade 6, so the measure was field tested and modified for grades 6 through 12. In year 1, 
we used the full MRQ scale. The MRQ had a Cronbach’s alpha of 0.93 in the fall and spring (50 
items). 


Impacts on Student Reading Achievement 


Sixth Grade. Table 1.12 shows the overall impact of the targeted intervention on sixth- 
grade students’ reading achievement and the impact broken out by special education status. The 
HLM results indicate no significant difference in overall achievement between students in the 
treatment and control groups (ES = 0.077, p = .137). There also is no significant difference in 
achievement of non-special education students between treatment and control groups (ES = 
0.076, p = .238). Finally, there is no significant difference in achievement between the special 
education students in treatment and control groups (ES = 0.099, p = .329). 
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Table 1.12 


Overall Impact of the Target Intervention on Student Achievement, and by Special Education 
Status, Sixth Grade 


HLM-adjusted 


Unadjusted Means Estimated Effect 


Means Impact Size , 
Control T™ Control T™ P 
All Students 
6'" Grade Spring NCE 30.7 32.3 29.3 30.4 1.07 0.077 .137 


(13.86) (13.92) 
6" Grade Students 530 605 
No. of Schools = 12 
Non-Special Education Students 
6" Grade Spring NCE 33.2 34.2 322 33.2 1.02 0.074 .238 
(13.50) (13.00) 
6'" Grade Students 379 404 
No. of Schools = 12 


Special Education Students 


6" Grade Spring 24.5 25.4 25.6 26.9 1.26 0.091 329 
NCE (12.79) (13.87) 
6'" Grade Students 151 201 


No. of Schools = 12 


Note. Standard deviations are presented in parenthesis. Effect size calculated as the impact divided by the full sample control 
group standard deviation. 


Appendix C (Exhibit Table 1) shows a summary of the full model achievement results for all 
intervention and control students in the sixth grade. Special education status is significant, 
indicating lower achievement for students in special education classes than other students. There 
is, however, very little explained variance in student achievement due to the effect of the school. 
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Appendix C (Exhibit Table 2) shows a summary of the full model achievement results for 
sixth-grade non-special education students. Ethnicity is significant, indicating that minority 
student achievement is lower than white student achievement. There is also very little explained 
variance in student achievement due to the effect of the school. 


Appendix C (Exhibit Table 3) shows a summary of the full model achievement results for 
sixth-grade special education students. There are no significant student variables and little 
explained variance in student achievement due to the effect of the school. 


Ninth Grade. Table 1.13 shows the overall impact of the targeted intervention on ninth- 
grade students’ reading achievement, and the impact broken out by special education status. The 
HLM results indicate a significantly larger average achievement score for all students in the 
treatment group than for students in the control group (ES = 0.122, p = .032). The difference 
between treatment and control groups is even greater for non-special education students (ES = 
0.204, p = .003). However, there is no significant difference in achievement between the special 
education students in the treatment and control groups (ES = 0.065, p = .549). This means the 
intervention may have been more beneficial for students who were not receiving special 
education services than for students who were receiving special education services. 
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Impact of the Target Intervention on Student Achievement of Non-Special Education and Special 


Education Students, Ninth Grade 


Unadjusted HLM-adjusted 
Means Means 
Control Tx Control Tx Estimated Effect p 
Impact Size 
All Students 
9" grade 32.2 33.7 30.2 31.9 1.69 0.122 .032 
Spring NCE (13.83) (14.87) 


9"" grade students 535 


No. of schools = 11 


9"" grade 33.4 
Spring NCE (13.12) 


9" grade students 411 


9" grade 28.7 
Spring NCE (15.50) 


9" grade students 124 


No. of schools = 11 


593 


Non-Special Education Students 


36.3 32.6 35.3 2.68 0.194 .003 
(13.69) 


417 


Special Education Students 


27.5 28.1 27.1 <1;01 -0.073. 549 
(15.75) 


176 


Note. Standard deviations are presented in parenthesis. Effect size calculated as the impact divided by the full sample control 


group standard deviation. 


Appendix C (Exhibit Table 4) shows a summary of the full achievement model results for 
ninth grade. Special education status is significant, indicating lower achievement for students in 
special education classes. There is very little explained variance in student achievement due to 


the effect of the school. 
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Appendix C (Exhibit Table 5) shows a summary of the full model achievement results for 
ninth grade who are not in special education. There are no significant student variables, and little 
explained variance in student achievement due to the effect of the school. 


Appendix C (Exhibit Table 6) shows a summary of the full model achievement results for 
ninth grade who are in special education. There are no significant student variables, and there is 
little explained variance in student achievement due to the effect of the school. 

Impacts on Students’ Reading Strategy Use 

Sixth Grade. Table 1.14 shows the impact of the LSC intervention on sixth-grade students’ 
overall strategy use. The HLM results indicate a marginally significant difference in reading 
strategy use for students receiving the treatment than for students in the control group (ES = 
0.129, p = .057). 

Table 1.14 


Overall Impact of the Target Intervention on Reading Strategy Use 


Unadjusted HLM-adjusted 
Means Means 
Control Tx Control Tx Estimated Effect p 
Impact Size 
6" grade spring 2.86 2.92 2.88 2.57 0.086 0.129 .057 
MARSI score (0.668) (0.711) 
6" grade students 390 439 
No. of schools = 12 
9" grade spring 2.64 2.73 2.78 2.85 0.068 0.091 .186 
MARSI score (0.745) (0.777) 
9" grade students 342 368 


No. of schools = 11 


Note. Standard deviations are presented in parenthesis. Effect size calculated as estimated impact divided by control group 
standard deviation. 


Appendix C (Exhibit Table 7) shows a summary of the full model reading strategy results for 


all intervention and control students in the sixth grade. Gender is a significant variable, indicating 
lower strategy use for males than females. Also, SES is significant, indicating that students 
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receiving free or reduced lunches use strategies more than students of higher SES. There is very 
little explained variance in student achievement due to the effect of the school. 


Ninth Grade. Table 1.14 shows the impact of the targeted intervention on ninth-grade 
students’ overall reading strategy use. The HLM results indicate there is no significant difference 
in reading strategy use between students in the treatment and control groups (ES = 0.091, p = 
.186). 


Appendix C (Exhibit Table 8) shows a summary of the full model reading strategy results for 
all treatment and control students in the ninth grade. Gender is a significant variable, indicating 
lower strategy use for males than females. Ethnicity is significant, indicating higher strategy use 
for minority students. Also, SES is significant, where students receiving free or reduced lunches 
report using strategies less than students of higher SES. There is very little explained variance in 
student achievement due to the effect of the school. 


Impacts on Student Motivation 
Sixth Grade. Table 1.15 shows the overall impact of the targeted intervention on sixth 
grade students’ reading motivation as measured by the MRQ. The HLM results indicate a 


significant difference in reading motivation for students in the treatment and control groups (ES = 
0.159, p= .016). 
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Table 1.15 


Overall Impact of the Target Intervention on Student Motivation 


Unadjusted HLM-adjusted 
Means Means 
Control Tx Control Tx Estimated Effect p 
Impact Size 

6" grade spring 2.65 2.71 2.65 2.73 0.075 0.159 .016 
MRQ score (0.466) (0.490) 
6" grade students 390 439 
No. of schools = 12 
9" grade spring 2.38 2.50 2.42 2.54 0.119 0.230 .001 
MRQ score (0.517) (0.534) 
9"" grade students 342 368 


No. of schools = 11 


Note. Standard deviations are presented in parenthesis. Effect size calculated as estimated impact divided by control group 
standard deviation. 


Appendix C (Exhibit Table 9) shows a summary of the full model motivation results for all 
intervention and control students in the sixth grade. There are no significant student variables in 
this model, and little explained variance in student achievement due to the effect of the school, 
with a small intraclass correlation. 


Ninth Grade. Table 1.15 shows the impact of the LSC targeted intervention on students’ 
reading motivation as measured by the MRQ. The HLM yields a significant difference between the 
motivation for intervention students and students in the control group (ES = 0.230, p = .001). 


Appendix C (Exhibit Table 10) shows a summary of the full model motivation results for all 
treatment and control students in the ninth grade. Gender is a significant variable, indicating 
lower motivation for males than females. Ethnicity is also significant, indicating a higher 
motivation for minority students. There is very little explained variance in student achievement 
due to the effect of the school. 
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Evaluation of the Implementation of the Whole-School Intervention 


The research questions that guided the implementation study of the whole-school 
intervention are: 


e What was the level of implementation of professional development support/participation 
for teachers? 


e What was the level of implementation of professional development support/participation 
for ALM coaches? 


e What was the level of implementation of professional development support/participation 
for administrators? 


Exhibit 4 indicates the data sources that were used to answer these questions. 
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Exhibit 4. 


Years 1 - 4 Data Sources on Implementation: School-Wide Intervention (CTL Adolescent Literacy 
Model) 


Measures/Data Sources 


Responsible Partner Record Review 
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Type and amount of 
professional 
development provided e e 
to teachers and level of 
participation. 


Type and amount of 
professional 

development provided e e e 
to administrators and 
level of participation. 


Type and amount of 
professional 
development provided e e e e 
to ALM coaches and 
level of participation. 


Type and amount of 
coaching provided to 
ALM coaches and level 
of participation. 


Percentage of class time 
teachers used the 
whole-school 
instructional practices. 


Whole-School Professional Development Inputs 


Table 2.1 shows the participation of content teachers in the ALM whole-school 
professional development inputs, and Table 2.2 shows the participation of the ALM coaches and 
administrators. In years 1 through 4, the percentage of content teachers who attended at least 1 
day of the whole-school summer training was calculated. There was a decrease in content 
teachers’ participation from year 1 to year 2, but attendance remained stable in years 3 and 4. In 
year 3, nine schools had more teachers participate in training than the number of certified 
teachers in the building and this occurred in eight schools in year 4. At the onset of the project, it 
was expected that at least 80% of content teachers would participate in the whole-school 
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professional development training. Table 2.1 shows the proportion of schools that achieved full 
participation in the professional development each year of the project. 
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ALM Implementation of Professional Development for Content Teachers 


School 
Level 


MS 
HS 
6-12 


MS 
HS 
6-12 


MS 
HS 
6-12 


MS 
HS 
6-12 


ALM PD for content teachers 


N % % at Full 
Attending at Attending® Participation 
Least One Day*® 


Year 1 
329 86.1 70 
409 83.8 66.7 
42 100 100 
Year 2 
330 84.5 80 
448 80.9 55.6 
39 78.0 50 
Year 3 
318 100.6 70 
439 89.1 66.7 
40 117.6 100 
Year 4 
289 91.5 70 
412 83.7 66.7 
58 128.9 100 


Note. In Year 1, teachers participated in 5 days of professional development. In Years 2, 3 and 4, teachers participated in 2 days of 


professional development. 


*In year 2, one school had more teachers participate in training than there were certified teachers on staff. In year 3, nine schools 
had more teachers participate in training than there were certified teachers on staff. In year 4, eight schools had more teachers 
participate in training than there were certified teachers on staff In all years, 280% of certified staff attending was considered 


complete participation in the training. 
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School administrators were expected to attend administrator training sessions on the ALM 
school-wide model each year of the project. Table 2.2 reflects a relatively high participation rate 
among administrators in these training sessions. Administrators were also expected to attend the 
school-wide trainings with their content teachers. 


In years 1 through 4, ALM coaches were expected to attend training sessions to learn 
about how to support the ALM whole-school model. Table 2.2 indicates that all literacy coaches 
participated in the professional development fully. They were assisted in this by CTL mentor 
coaches who individually provided missed content to coaches who were absent from any part of 
the training. 


In addition to these training sessions, mentor coaches from CTL provided on-site and 
distance support to ALM coaches. The numbers of days of on-site support provided for each year 
of the project are reported in Table 2.2. Mentor coaches also provided support for ALM coaches 
as needed, through phone calls, emails, and electronic distance support. This support was for 
both the targeted intervention and whole-school intervention, and although documentation of 
this support was provided for years 2 through 4, the extent to which the support was specific to 
the targeted intervention or whole-school model was not designated in year 1. Overall, ALM 
coaches and mentor coaches engaged in 10,461 minutes of phone support calls, exchanged 2,410 
emails, and participated 357 times in formal distance support in year 2. In year 3, LSC teachers 
and mentor coaches engaged in 11,490 minutes of phone support calls, exchanged 4,320 emails, 
and participated 45 times in formal distance support. In year 4, LSC teachers and mentor coaches 
engaged in 433 phone support calls, exchanged 2,791 emails, and participated 57 times in formal 
distance support. 


Many literacy coaches completed hallmark assignments as part of their certification 
program through U of L. In year 1, 19 literacy coaches participated, in years 2 and 3, 16 
participated, and in year 4, 15 participated. By 2010, 11 teachers had finished the certification 
program and graduated. Two more teachers are on track to complete their certification and 
degree by the end of 2011. 
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Table 2.2 


Means for ALM Implementation of Professional Development for Administrators and ALM 
Coaches 


ALM PD for ALM PD for ALM Mentor 
Administrators® Coaches? Coaching 
Mean Mean 
Days Days Mean Site 
Attended % at Full Attended % at Full Visit Days 
(Min,Max) n Participation’ (Min,Max) n Participation (Min,Max) 
Year 1 
MS 1.7 10 90 11.1 10 100 5.4 
(1,2) (8,12) (4.2,7.5) 
HS 1.7 12 77.8 10.6 9 100 6.4 
(.5,2) (7,12) (2.5,12) 
6-12 1.8 2 100 12.0 2 100 6.4 
(1.5,2) (5.3,7.5) 
Year 2 
MSs 1.3 10 100 9.0 10 100 5.6 
(1,1.5) (8.5,9.5) (3,8) 
HS 1.3 12 100 9.1 9 100 5.1 
(1,1.5) (7.5,9.5) (3,8) 
6-12 1.5 2 100 9.5 2 100 6.0 
(5,7) 
Year 3 
MS 1.3 10 100 8.4 10 100 5.9 
(1,1.5) (8,9) (3,11) 
HS 1.3 12 100 7.8 9 100 5.3 
(1,1.5) (6.5,9) (3.5,8) 
6-12 1.5 2 100 8.5 2 100 5 
(8,9) (3,7) 
Year 4 
MS 1.4 10 100 6.1 10 100 4.4 
(.5,1.5) (4,6.5) (3,6) 
HS 1.4 12 100 5.3 9 100 6.2 
(1,1.5) (3,6.5) (3,10) 
6-12 1.5 2 100 6.0 2 100 6 
(5.5,6.5) 


“Administrators had specific training days. In year 1, there were 2 days of PD. In years 2, 3 and 4 there were 1.5 days of PD. 
Administrators were encouraged to also attend the ALM PD intended for content teachers. In year 1, 21 administrators from 12 
schools attended at least 1 day. In year 2, 16 administrators from 12 schools attended at least 1 day. In year 3, 21 administrators 
from 17 schools attended at least 1 day. In year 4, 21 administrators from 21 schools attended at least 1 day. 

"In year 1, ALM coaches had 12 days of PD; year 2, 9.5 days of PD; year 3, 9 days of PD; and year 4, 6.5 days of PD. ALM coaches 
that missed group training sessions received make up training from the mentor coaches. 

“Participation for year 1 was complete with 21.5 days attended, and years 2, 3 and 4 were complete with 2 1 day attended. 
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Classroom Observations. To judge content teachers’ rate of use of ALM strategies in their 
classrooms, evaluators observed selected content teachers twice each year in each year of the 
project. During the 2006-2007 school year, all E/LA teachers in grades 6 and 9 were observed 
twice in the spring (n = 78 observations). During the 2007-2008 academic year, all sixth- and 
ninth-grade Language Arts teachers were again observed twice, once in the fall and once in the 
spring (n = 101 observations). Additionally, two middle schools and two high schools were 
randomly selected for additional observations in math, science and social studies classrooms (n = 
49 observations). Those content teachers were also observed twice, once in the fall and again in 
the spring. During the 2008-2009 and the 2009-2010 academic year, a random sample of content 
teachers were observed twice, once in the fall and once in the spring (n = 112 observations). With 
each observation, an interview was conducted with the teacher to provide further clarity for the 
observer. 


Observers were trained through a multiphased process that involved both the model 
developers and lead researchers. In the summer of 2006, research assistants attended a training 
session, conducted by CTL trainers, along with the content area teachers in the Striving Readers 
schools. Following that training, the research assistants and lead researchers worked 
collaboratively to create an observation protocol. First, they listed the six instructional domains 
present in the school-wide intervention (fluency, comprehension, writing to use what you know, 
writing to learn, academic dialogue, vocabulary development). Second, they identified activities 
associated with each component. This section of the observation protocol provided evidence as 
to whether any of the six instructional domains were present during content area instruction. 
Interview questions were constructed to supplement observations. The investigators then 
constructed the observation protocol and sent it to the Professional Development Director at CTL 
for feedback. Codes were developed to characterize the range of instructional behaviors observed 
in the intervention classrooms. 


The investigators met with the research assistants to provide training related to taking field 
notes. Training consisted of lecture related to taking field notes, watching video segments, 
practice taking field notes, and critique. Research assistants were trained to organize their field 
notes in 5-minute time intervals, capturing as much detail about instruction and classroom 
dialogue as was possible. Sample field notes were shared, critiqued, and refined. 


In November of 2006, research assistants went out in pairs and practiced taking field notes 
in three content area classrooms. In December of 2006, the group reconvened and research 
assistants practiced coding the data and discussing the codes after each observation was coded. 
Each 5-minute segment was coded using codes to represent (a) the ALM domains (fluency, 
comprehension, writing to use what you know/writing to learn, vocabulary development, and 
academic dialogue), (b) Other Literacy Activities, (c) Non-Literacy Activities, or (d) Behavior 
Management Activities. Agreement was discussed but not compiled/computed at this meeting. 


After the observations were conducted in year 1, four members of the evaluation team 
sought inter-rater reliability for coding the observation protocols. First, two coprincipal 
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investigators coded and discussed one protocol. They used that coding event to establish initial 
rules for coding. Second, they coded five protocols independently. They discussed those five 
coded protocols and refined the coding rules further. Third, they recoded the initial five protocols 
using the new rules. Next, they coded an additional five protocols using the new rules. Inter-rater 
agreement on all 10 protocols (13% of the data) was 92%. Agreement on the last five protocols 
was 82%. After inter-rater agreement was established, raters discussed and came to 100% 
consensus on all codes. 


The investigators used three of the coded protocols and three additional protocols to train 
two advanced doctoral students in literacy to use the coding rules. The investigators coded eight 
additional protocols (four each) and both doctoral students coded all eight protocols. Inter-rater 
agreement between the two doctoral students and each of the coprincipal investigators was 
83.1% and 89.8% respectively. Overall inter-rater agreement across all raters was 85.5%. 


Classroom Implementation Results. Table 2.3 shows the mean percentages of content 
class time that content-area teachers implemented the ALM whole-school intervention 
techniques. In year 1, the sixth- and ninth-grade E/LA teachers spent approximately one quarter 
of their class time using the whole-school intervention techniques. In year 2, the table indicates 
that use of the whole-school intervention techniques decreased among ninth-grade teachers and 
increased slightly among sixth-grade teachers. It is important to note that the composition of 
content area teachers changed in year 2 to include other content area teachers, but most (26 
teachers) were E/LA teachers. In year 3, both sixth- and ninth-grade teachers slightly increased 
the percent of time using whole-school techniques. However, the composition of content area 
teachers observed had again changed to include a smaller number of E/LA teachers (8 sixth grade, 
and 6 ninth grade). In year 4, the ninth-grade teachers increased the percent of time using whole- 
school techniques. It is interesting that, for all years of the program, middle school teachers 
exhibited higher levels of implementation than high school teachers, overall. The ALM school- 
wide intervention trainer indicated that at least 30% of content-area teachers’ class time should 
be spent on the school-wide techniques, and this level of implementation was achieved by sixth- 
grade teachers in years 2, 3, and 4. 
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Table 2.3 


Implementation of the ALM Model in Content Classrooms as Measured by Percent of Class Time 


Year 1° Year 2° Year 3° Year 4° 

6" grade 
Observed ALM 25.7% 30.1% 32.2% 34.1% 
Number of teachers observed 25 39 30 27 
Number of observations 42 77 58 52 

gin grade 
Observed ALM 24.3% 13.3% 16.1% 23.5% 
Number of teachers observed 24 38 27 28 
Number of observations 36 73 54 54 


Note. In year 1, teachers were observed in the spring semester. In years 2 and 3, teachers were observed once in the fall and 

once in the spring. 

“all teachers observed were English/Language Arts teachers in year 1. 

"Twenty-six of the teachers observed in the sixth grade and 26 of the teachers observed in the ninth grade were 

English/Language Arts teachers in year 2. 

“Eight of the teachers observed in the sixth grade and six of the teachers observed in the ninth grade were English/Language 

Arts teachers in year 3. 

“Five of the teachers observed in the sixth grade and seven of the teachers observed in the ninth grade were English/Language Arts 
teachers in year 4. 


Implications for the Impact Analysis. In effectiveness studies, it is essential to examine the 
extent to which the intervention was implemented with fidelity so that appropriate conclusions 
may be drawn from the research findings (Durlak & DuPre, 2008; Institute of Education Sciences, 
2009). If student outcomes do not improve after participation in an intervention, one of two 
explanations may be attributed: (a) the intervention, as designed, is not effective, or (b) the 
intervention was not implemented as it was designed or as the developer intended. Participation 
in the professional development inputs was very high for literacy coaches and relatively high for 
content teachers across all years of the project. Implementation of the classroom model was 
higher for middle school teachers than for high school teachers. Although the ideal level of 
classroom implementation of the whole-school model has not been empirically determined, it 
might be expected that if the whole-school model is effective, then middle schools will yield 
greater impacts than high schools. 


Evaluation of the Impacts of the Whole-School Intervention 


The research questions that guided the evaluation study of the whole-school ALM 
intervention are: 


e What is the impact of the ALM on teachers’ personal and collective efficacy? 
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e What is the impact of the ALM on student achievement in reading and writing? 
Description of the Teacher Samples 


Impacts of the whole-school intervention were evaluated by comparing teacher outcomes in 
Striving Readers and matched comparison schools. These matched pairs of schools were identified 
by the Kentucky Department of Education (KDE) based on ethnicity, number of students, percent 
free and reduced lunch, and accountability index. Of the 21 matched schools identified by the KDE, 
12 schools participated in the Teacher Efficacy Survey in year 1, 9 schools in year 2, 8 schools in year 
3, and 4 schools in year 4. 


Demographic data on teachers was gathered from the Teacher Efficacy Survey (Table 2.4). 
Teacher demographics at the Striving Readers and matched schools are similar with respect to 
gender, ethnicity, and education level. The corresponding pie charts (Figure 1.1) show content areas 
were represented with similar proportions of teachers in the Striving Readers and matched schools 
across all 4 years. Striving Readers schools had a larger proportion of middle school teachers, and a 
smaller proportion of high school teachers complete the questionnaire as compared to the matched 
schools (Figure 1.2). Striving Readers school teachers completing the questionnaire had slightly less 
experience, with over half with 7 or fewer years teaching experience (Figure 1.3). 


Table 2.4 


Teacher Demographics for Striving Readers and Matched Schools, Years 2006-2010 
Gender Ethnicity Highest Degree 
No. of BA/ No. 
Surveys F M Wht. Blk. Othr BS MA Spec. Doc. Rsp. 


Striving Readers Schools 


No. 1295 851 444 1237 31 27 276 719 262 15 23 
(Prop.) (.66)  (.34) (.96) (.02) (.02) (.18) (52) (.26) (.02)  (.02) 
Matched Schools 
No. 534 350 184 496 22 16 97 279 137 8 13 
(Prop.) (.65)  (.35) (.93) (.04) (.03) (.21) (.56) (.20) (.01) (.02) 
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Figure 1.1. Striving Readers and Matched Schools Teachers’ Primary Content Responsibility. 


Matched Schools Striving Readers Schools 


PrimarySubject 
Gi Language Arts/Speech 
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Figure 1.2. Striving Readers and Matched Schools Teachers’ School Level. 
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Figure 1.3. Striving Readers and Matched Schools Years of Teacher Experience. 


Matched Schools Striving Readers Schools 


Experience 


Teacher Efficacy Measure 


The Teacher Efficacy Survey was used to determine the impact of the whole-school 
professional development model on teachers’ personal and collective efficacy for literacy teaching. 


Teacher Efficacy Survey. Teachers’ sense of efficacy for teaching has been associated with 
effective classroom practices (Ashton & Webb, 1986; Gibson & Dembo, 1984) and higher student 
achievement (Ross, 1992). At the start of the project and each subsequent year, surveys of teacher 
efficacy were used to determine the effects of the project on teachers’ sense of efficacy for literacy 
teaching. A teacher efficacy survey comprised of 60 items to measure Personal Teaching Efficacy 
(PTE) and Collective Teacher Efficacy (CTE) was administered to literacy coaches. PTE items were 
drawn from teacher efficacy instruments developed by Woolfolk and Hoy (1990), Hoy and Woolfolk 
(1993), and Gibson and Dembo (1984). PTE items include statements, such as, “Some students are 
not going to make a lot of progress this year in reading, no matter what | do.” CTE items were 
developed by Goddard, Hoy, and Hoy (2000) and include statements, such as, “If a child does not 
want to read in their content area, most teachers in my school give up.” Some questions from the 
original surveys were altered to reflect more of a reading emphasis (e.g., “When a student does 
better than usual in reading, it is often because | exerted a little extra effort.”), and some additional 
questions were added by the evaluation team that focused specifically on processes related to 
teaching content area literacy, such as, “I know how to teach vocabulary effectively.” All items used a 
6-point Likert-type format, ranging from 1 is equal to Strongly Agree to 6 is equal to Strongly 
Disagree. 


All of the original instruments from which the present survey was adapted have 
demonstrated high reliability and validity. Gibson and Dembo (1984) reported Cronbach’s alpha 
coefficient of 0.75 on the PTE subscales. Goddard et al. (2000) reported Cronbach’s alpha 
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coefficient of 0.96 for the CTE subscale. In the present study the reliability of each subscale, with the 
revisions described above, was evaluated for content teacher survey results. The PTE subscale was 
calculated each year and shown to be reliable, with lowest results in the summer of 2006 of (a = 
0.878, n= 624). Likewise, the CTE subscale was reliable, with lowest results also shown in the 
summer of 2006 of (a = 0.801, n = 650). 


Impacts on Teacher Efficacy 


The school-wide intervention teachers’ efficacy was measured using the Teacher Efficacy 
Survey. Data were gathered prior to training in the summer of 2006 and again in the summers of 
2007 through 2010. Table 2.5 displays the means, standard deviations and sample sizes of Striving 
Readers and matched comparison content area teachers’ self-reported personal and collective 
efficacy. Each year more teachers from Striving Readers schools participated in the survey than from 
matched schools, and the number of matched school teachers declined every year. In 2006, 30% of 
the survey participants were from matched schools. However, by 2010, fewer than 15% of the 
teachers participating were from matched schools. 
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Table 2.5 


Means and Standard Deviations on Personal and Collective Efficacy Subscales of the Teacher 
Efficacy Survey for Striving Readers and Matched Schools Across Time 


Personal Efficacy Collective Efficacy 

Summer Striving Readers Matched Striving Readers Matched 
2006 

n 737 319 735 316 

M 3.82 3.87 4.02 4.18 

SD (.58) (.53) (.67) (.68) 
2007 

n 724 208 718 208 

M 3.98 3.96 4.22 4.15 

SD (.51) (.52) (.63) (.65) 
2008 

n 632 202 627 206 

M 3.98 3.91 4.22 4.07 

SD (.52) (.62) (.62) (.69) 
2009 

n 734 251 729 251 

M 4.14 4.03 4.41 4.26 

SD (.52) (.55) (.58) (.69) 
2010 

n 696 121 695 121 

M 4.04 4.05 4.32 4.25 

SD 55 55 .63 65 
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The personal efficacy averages over time are illustrated in Figure 2.1, and the collective 
efficacy averages are illustrated in Figure 2.2. Figure 2.1 shows that Striving Readers teachers had 
slightly lower personal efficacy in the summer of 2006 than the teachers at the matched schools. 
However, by the summer of 2008, the Striving Readers teachers had slightly higher personal efficacy 
than the matched school teachers. Striving Readers teachers and matched school teachers’ 
increased in personal efficacy in 2009, and the difference between the two groups increased. By 
2010, personal efficacy for teachers in Striving Readers schools dropped to the level of the matched 
schools. 


Figure 2.1. Means of Teachers’ Self-Reported Personal Efficacy as a Function of Time and Treatment 
Condition. 


== Striving Readers 
== Matched 


Mean Personal Efficacy 


2006 2007 2008 2009 2010* 


Year 


*Note: Only four matched schools participated in 2010. 


A somewhat similar trend can be seen in Figure 2.2, showing that teachers in the Striving 
Readers schools began with lower collective efficacy than the teachers at matched schools and by 
the summer of 2008 had higher collective efficacy. However, a fairly steep decline in collective 
efficacy of teachers at matched schools between 2007 and 2008 is evident. This decline was not 
shown as sharply in the personal efficacy data. Collective efficacy for teachers in Striving Readers 
and matched schools increased in 2009. In 2010, the collective efficacy of teacher in Striving Readers 
schools dropped to the level of the matched schools, which was also shown for personal efficacy. 
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Figure 2.2. Means of Teachers’ Self-Reported Collective Efficacy as a Function of Time and Treatment 
Condition. 


== Striving Readers 


=—™@= Matched 


Mean Collective Efficacy 


*Note: Only four matched schools participated in 2010. 


Changes in Teacher Efficacy for Literacy Coaches (LSC Teachers/ALM Coaches) 


The literacy coaches’ self efficacy was measured using the Teacher Efficacy Survey. Pretest 
data were gathered prior to training in the summer of 2006, or at the time new literacy coaches were 
hired to fill vacancies. Literacy coaches’ efficacy was measured each subsequent summer. Table 2.6 
displays the means and standard deviations for literacy coaches on the Personal Efficacy and 
Collective Efficacy Subscales by year of exposure to the Striving Readers program. This table, along 
with Figure 2.3, illustrates that literacy coaches decreased in their sense of personal efficacy after 
their first year in the program. After participating in Striving Readers for a second year, literacy 
coaches’ personal efficacy for literacy teaching rebounded and surpassed their initial efficacy level. 
Personal efficacy remained steady after a third year in the program, and increased after a fourth 
year. Conversely, literacy coaches’ sense of collective efficacy increased after one year of 
participation in the Striving Readers program, remained steady after a second year, and increased at 
the end of a third and fourth year. 
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Table 2.6 
Literacy Coach Efficacy by Exposure to the Striving Readers Program 
n Minimum Maximum M SD 


Personal efficacy 


At time of hire 23 3.1 5.3 4.40 635 
After first year 17 3.0 5.3 4.34 695 
After second 26 3.7 5.3 4.65 AT74 
year 

After third year 18 2.3 5.5 4.64 717 
After fourth 13 4.4 5.3 4.90 314 
year 


Collective efficacy 


At time of hire 23 2.7 4.7 3.73 529 
After first year 17 2.4 5.1 3.96 .790 
After second 26 2.4 5.4 3.92 756 
year 

After third year 18 2.4 5.2 4.02 .678 
After fourth 13 3.1 5.2 4.14 595 
year 


“Literacy coaches were given survey at the time of hire and each subsequent summer in the program. Due to turnover and 
uncompleted questionnaires, the number of completed questionnaires varies by year. 
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Figure 2.3. Literacy Coach Efficacy by Exposure to Striving Readers Program. 


=@=— Personal 
== Collective 


1 2: 3 


Mean Efficacy by Years of Exposure 


Measures of Student Outcomes 


The KCCT is a criterion-based exam given to all Kentucky students each spring and is an 
important component of the Kentucky Testing System (KDE, 2011). The KCCT assesses student 
achievement in the core content areas, including reading, writing, mathematics, science, and social 
studies. A composite score by content area is derived annually for every school in Kentucky. These 
scores are used to evaluate the impact of the ALM whole-school intervention. It is important to note 
that the criteria for scoring the KCCT results changed in 2007, and this change resulted in higher 
scores than did the previous scoring criteria. The timing of this change coincides with the first year 
of the ALM implementation. 


ALM implementation began in the fall of 2006. Because most KCCT tests are given in the 
spring, the 2006 spring KCCT scores are preintervention scores and the 2006-07 academic year is the 
first year of implementation. High school writing KCCT exams, however, are given in the fall of each 
year and are published in the following spring. So, a high school writing test taken in the fall of 2006 
is a preintervention score but will be published as a 2007 score. Consequently, the intervention year 
will be shown as 2007 through 2008 for the high school writing scores. 


Summary of Analytic Approach 


A short interrupted time-series quasi-experiment design (SITS) has observations of values, 
such as school test results, collected at specific time intervals. At some point in the time-series, an 
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intervention would have been introduced that may effectively “interrupt” the direction of the time 
series. The change, or “jump” at the introduction of the intervention, and the change in slope is 
measured to determine the impact of the intervention. A “Short Interrupted Time-Series with 
Multiple Cases Each Measured Repeatedly” (SITSMCMR) is the quasi-experimental design used to 
measure the impact of the whole-school ALM intervention (Braver & Brahm, 2005). This method 
included multiple measures of Striving Reader school KCCT scores 4 years prior to and 4 years after 
the ALM intervention. In addition, the model includes KCCT scores for matched school for the 
corresponding years. 


At the beginning of the project comparison schools were identified by the KDE based on 
ethnicity, number of students, percent free and reduced lunch, and accountability index. Data are 
available for 9 of 12 matched middle schools and 10 of 11 matched high schools. 


Impact Model for Short Interrupted Time Series with Multiple Cases each Measured 
Repeatedly (SITSMCR) with Comparison Schools 


Yi, = Bo + @, + A(t) + B,(Time__ BMM ,,) + £,(spline ;,) + 
£,(Irt, * spline ,,) + B,(Trt, * spline; * Time __BMM ,,) + € jx 
Where: 


B, + Qo, = Baseline mean projection for matched schools, 

fi, = Difference in baseline mean projection, Striving Readers versus matched schools, 
P, = Postinterruption slope, matched schools, 

/; = \nterruption year slope (“jump”), matched schools, 


£,= Interruption year impact; difference between Striving Readers predicted and observed difference 
and matched school predicted and observed difference in interruption year 1 (“jump”) 
Bb = Impact on slope; difference between Striving Readers and matched school post-interruption slopes 


& i, = error for j=4 preinterruption time periods, and k=4 postinterruption time periods, 
and 


Trt, = 1 for Striving Reader schools, 0 for matched schools 
Time _BMM ;, = 0 for preinterruption time period (spring 2003, ‘04, ‘05, ’06), and 
=0 for spring 2007, 1 for spring 2008, 2 for spring 2009, and 3 for spring 2010 
spline ,, = 0 for preinterruption time period (spring 2003, ‘04, ‘05, ’06), and 
=1 for postinterruption time period (spring 2007, 2008, 2009, and 2010) 


Impacts on Student Achievement 


The SITSMCMR model is used to measure the impact of the ALM on the student achievement 
at the Striving Readers schools. Inherent in the model is the assumption that the Striving Readers 
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schools would perform similarly to the matched schools had there not been an intervention. The 
behavior of the matched schools is shown by three estimates of the model. The first estimate is of 
the intercept or the baseline mean projection, which is a measure of how the matched schools would 
be expected to perform during the postintervention years. The second measure estimates the effect 
of the matched school postinterruption slope (estimate of the slope from spring of 2007 through 
spring of 2010). The final measure estimates the matched school slope during the first year of 
implementation; from spring of 2006 to the spring of 2007. 


There are three corresponding SITSMCMR measures to determine the impact of the ALM on 
the Striving Readers schools. The first measure estimates the effect of the treatment group on the 
baseline mean projection, or difference between Striving Readers baseline mean projection and the 
matched school projection. The second measure estimates the impact of the ALM during the 
interruption year, or the first year ALM was implemented, by measuring the difference between the 
“jump” in Striving Reader school scores between spring of 2006 and spring of 2007 and the change in 
matched school scores for the same year. The final measure estimates the impact on the slope after 
the ALM was implemented by estimating the difference between Striving Readers and matched 
school slopes from the spring of 2007 through 2010. 


Graphs depicting each school-by-school treatment group (Striving Readers or matched) are in 
Appendix D. 


Reading Achievement 


Middle school. The SITSMCMR approach first considers the trend in achievement for 
matched schools. The results of middle school SITSMCMR analysis of reading achievement are 
shown in Figure 2.4 and Table 2.7. As can be seen in both the figure and the table, the matched 
schools had a significant increase between 2006 and 2007. This “jump” for the matched schools was 
due to a change in the scoring criteria of the reading KCCT. 


Comparing the matched school with the Striving Readers school KCCT scores, there is a 
significant difference in baseline projections. The figure shows the Striving Readers school reading 
achievement is significantly lower prior to the intervention, which challenges the assumption of the 
SITSMCMR model. There are no significant differences between the Striving Readers and matched 
school model results postintervention. 
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Figure 2.4. Middle School Reading KCCT Scores, Striving Readers and Matched Schools Averaged by 
Year (Spring). 
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*The increase in reading scores (2006-07) was due to a change in the scoring criteria. 
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Table 2.7 


Middle School Reading Achievement SITSMCMR Results 


Difference of Impacts 


Effect Estimate SE df t Pr>|t| 
By + Qo, . Intercept 


90.0077 1.4792 16 60.85 <.0001 


ee 6.3661 2.1088 119 ~-3.02  0.0031* 
Pog ie EMM 0.3931 0.6542 119 0.6 0.5491 
Pa sPine 3.3804 1.9351 119 2.65 0.0092* 
ibs ey Vln 03959 2.7495 119 0.67 0.5025 


Le oe 
A; trtgrp*Time_B*Spline 9.9793 = 9.9251 «119~Ss«1.05 0.2955 


*Significant at the .01 level 


High school. The results of high school SITSMCMR analysis of reading achievement are shown 
in Figure 2.5 and Table 2.10. As can be seen in both the figure and the table, the matched schools 
had a significant increase between 2006 and 2007. Again, this “jump” for the matched schools was 
due to a change in the scoring criteria of the reading KCCT. 


Comparing the matched school with the Striving Reader school KCCT scores, there is no 
significant difference in baseline projections. The figure shows the Striving Reader school reading 
achievement is very similar to the matched school achievement prior to the intervention. There are 
no significant differences between the Striving Reader and matched school model results 
postintervention. 
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Figure 2.5. High School Reading KCCT Scores, Striving Readers and Matched Schools Averaged by 
Year (Spring). 
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*The sharp increase in scores in reading (2006-2007) was due to a change in the scoring criteria. 
Table 2.8 


High School Reading Achievement SITSMCMR Results 


Difference of Impacts 


Effect Estimate SE df t Pr > |t| 
Bo + 2 oq. Intercept 74.3687. 2.1127 18 e502 <.0001 
f, . trterp 1.2842 2.9878 136 0.43 0.668 
Bf, Time_BMM -0.657 0.7937. 136 ~=-0.83 0.4092 
A; , Spline 156663. 23477 136° ‘6.67 <.0001* 
f,. trtgrp*Spline 28448 3.3201 136 -0.84 0.4006 
Bs, 
trtgrp*Time_B*Spline 0.782 1.1224 136 0.7 0.4872 
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Writing Achievement 


Middle school. The results of middle school SITSMCMR analysis of writing achievement are 
shown in Figure 2.6 and Table 2.9. As can be seen in both the figure and the table, the matched 
schools had a significant increase between 2006 and 2007. This “jump” for the matched schools was 
due to a change in the scoring criteria of the reading KCCT. 


Comparing the matched school with the Striving Reader school KCCT scores, there is no 
significant difference in baseline projections. However, Figure 2.6 shows that the Striving Readers 
school averages form a positive, linear trend whereas matched schools do not. The disagreement of 
middle school Striving Reader and matched school scores prior to the intervention again challenge 
the assumption of the SITSMCMR analysis. There are no significant differences between the Striving 
Reader and matched school model results postintervention. 


Figure 2.6. Middle School Writing KCCT Scores, Striving Readers and Matched Schools Averaged by 
Year (Spring). 
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*The sharp increase in on-demand writing scores in 2006-2007 was due to a change in the scoring criteria. 
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Table 2.9 


Middle School Writing Achievement SITSMCMR 


Difference of Impacts 


Effect Estimate SE df t Pr > |t| 
By + Gq, Intercept 61.9441 2.6745 16 23.16  <.0001 
(en OU 6.1424 3.8184 119 ale © Osuloe’ 
B,, Time_BMM -0.8276 1.3002 119 0.64 0.5256 
B,. Spline 24.2915 3.8459 119 8.28  <,0001* 
B,, trtgrp*Spline ‘Linley 5.4641 119 0.46 0.6456 
Bs. 
trtgrp*Time_B*Spline 0.09182 1.8387 119 0.05 0.9603 


High school. The results of high school SITSMCMR analysis of writing achievement are shown 
in Figure 2.5 and Table 2.12. Recalling that the high school writing KCCT is given in the fall, the lag 
will show the first year of intervention and scoring changes in 2007 through 2008. As can be seen in 
both the figure and the table, the matched schools had a significant increase between 2007 and 
2008, which is again due to a change in the scoring criteria of the writing KCCT. 


Comparing the matched school with the Striving Reader school KCCT scores, there is no 
significant difference in baseline projections. The figure shows the Striving Reader school reading 
achievement is very similar to the matched school achievement prior to the intervention. There are 
no significant differences between the Striving Reader and matched school model results 
postintervention. 
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Figure 2.7. High School Writing KCCT Scores, Striving Readers and Matched Schools Averaged by 
Year (Spring). 
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*The sharp increase in scores in on-demand writing (2007-2008) was due to a change in the scoring criteria. 


Table 2.10 


High School Writing Achievement SITSMCMR Results 


Difference of Impacts 


Effect Estimate SE df t Pr> |t| 

By + aq, . Intercept 67.4394 2.2923 18 29.42 <.0001 
f,. trterp 0.414 3.2418 136 0.13 0.8986 
B,.Time_BMM 1.2966 1.0825 136 1.2 0.2331 
B, . Spline 7.8319 1.5562 136 5.03 <.0001* 
B,. trtgrp*Spline 2.8779 2.2008 136 1.31 0.1932 
-0.9022 1.5309 136 -0.59 0.5566 


f, , trtgrp*Time_B*Spline 
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Summary and Conclusions 


This study examined the implementation and impacts of the Danville, KY Striving Readers 
project on teachers and students in 21 rural middle and high schools. It evaluated the impact of a 
targeted intervention on students’ achievement, strategy use, and motivation, and it investigated 
the impact of a whole-school literacy model on teachers’ efficacy and student achievement. In 
addition, this report addresses the extent the program was implemented at the professional 
development and classroom levels. 


Over the course of this Striving Readers project, the professional development model was 
implemented as planned. Participants were provided extended professional development and 
training through workshops, on-site mentoring, and a literacy certification program for school 
literacy coaches. Literacy coaches, administrators, and content area teachers achieved high levels 
of participation in the professional development inputs, overall, for both the targeted intervention 
and whole-school model. 


However, there was more variation in the levels of classroom implementation of the 
targeted and whole-school interventions. For the targeted intervention, classroom 
implementation was relatively low in year 1 but improved to high levels in the final years of the 
project. For the school-wide model, implementation by middle school content teachers was 
consistently higher than implementation by high school teachers, with middle school teachers 
implementing at adequate levels consistently. 


In terms of project impacts, the Striving Readers project has yielded positive effects for 
students and teachers. The targeted intervention, the LSC, had a significant impact on ninth-grade 
students’ reading achievement, sixth-grade students’ reading strategy use, and sixth- and ninth- 
grade students’ reading motivation. Although analyses of student achievement results did not 
reveal impacts of the whole-school model on students’ reading and writing achievement, the 
project was successful in influencing teachers’ efficacy for literacy teaching in Striving Readers 
schools over the course of the project. The literacy coaches who implemented the LSC in 
supplemental classrooms and supported the implementation of the whole-school model in their 
buildings demonstrated increasingly higher efficacy after the initial year of implementation. 
Similarly, content area teachers who participated in Striving Readers schools increased their levels 
of personal and collective efficacy across the years of the project. 


Achievement results for the evaluation of the supplemental LSC course suggest the 
intervention is promising for improving the reading performance of students who are in their first 
year of high school. Analyses indicate that this intervention had a positive impact on ninth-grade 
students’ reading achievement overall. A closer look at ninth-grade students’ achievement 
revealed that the significant impacts of the LSC were evident for students who were not served by 
special education. Impacts were not significant for ninth-grade students who were served by 
special education. This suggests that the LSC is most effective for students who do not have 
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disabilities and that it may be less effective in improving the achievement of students who do have 
disabilities. 


Although similar achievement results were not noted for sixth-grade students in this 
evaluation, the LSC did have a significant impact on sixth-grade students’ reading strategy use at 
the < .10 level. Students in the LSC class reported using cognitive reading strategies to a greater 
extent than students in the control group, which suggests that the LSC strengthens students’ 
knowledge of reading strategy use at this phase of schooling. It is interesting that this impact did 
not transfer to improved reading achievement for sixth-grade students because students’ use of 
cognitive reading strategies is associated with reading achievement (Alexander & Murphy, 1999; 
Baker & Brown, 1984; Paris, Lipson, & Wixon, 1983; Pressley & Afflerbach, 1995; Pressley, 2000). 
Similarly, it is interesting that the LSC did not have an impact on ninth-grade students reported use 
of strategies given that the intervention did have a positive impact on students’ reading 
achievement. 


A strong and consistent finding of this study, for both sixth- and ninth-grade students, is 
the significant impact of the LSC class on students’ reading motivation, overall. Recommendations 
for improving adolescents’ literacy performance tend to include some focus on reading motivation 
(Biancarosa & Snow, 2004; Alvermann, 2001; Kamil et al., 2008), so it is important to note the 
effectiveness of the supplemental LSC class in for influencing middle and high school students’ 
reading motivation in positive ways. Research suggests that more motivated readers read more 
and reach higher levels of achievement (Organisation for Economic Co-operation and 
Development, 2009), but the increase in reading motivation did not necessarily result in improved 
reading achievement in this evaluation, at least for sixth grade. However, it is possible that sixth- 
grade students’ increased motivation will pay off in terms of improved reading achievement over 
time. 


The design of this study provides a number of new insights regarding the LSC. First, the 
study shows promising results for examining the impact of the LSC as a set of coherent strategies 
rather than studying the impact of each component individually, as has been done in previous 
research (Bulgren, Hock, Schumaker, & Deshler, 1995; Clark, Deshler, Schumaker, Alley, & Warner, 
1984; Lee & Von Colln, 2003; Lenz & Hughes, 1990; Nagel, 1982; Schumaker & Deshler, 1992; 
Woodruff, Schumaker, & Deshler, 2002). Second, findings from this study suggest that the LSC has 
positive benefits related to reading achievement, strategy use, and motivation, particularly for 
students who do not participate in special education. Finally, this study provides empirical 
evidence about the impact of the LSC using a randomized pretest-posttest control group design 
with larger numbers of students than had been available in previous studies. 


In addition to evaluating the LSC, this study evaluated the impact of ALM, a whole-school 
literacy model designed to help teachers integrate literacy strategies in the content areas. 
Interrupted time series analyses of KCCT achievement data did not suggest impacts of the whole- 
school model on students’ reading and writing achievement. These results were not surprising, 
given how difficult it is to link professional development models for teachers directly to student 
achievement outcomes. However, the whole-school literacy model did have positive impacts on 
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content teachers’ sense of efficacy for literacy teaching. Teachers’ self-efficacy for teaching has 
been associated with effective classroom practices and higher student achievement (Ashton & 
Webb, 1986; Gibson & Dembo, 1984; Ross, 1992), so it is likely that this increase in efficacy for 
teachers resulted in benefits for students not captured by our measures. 
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Literacy Coach Professional Development: 
4-day Literacy Coach Institute with 2 days 
dedicated to targeted intervention (Strategic 
Instruction Model — SIM) and 2 days dedicated 
to school wide content literacy model. All days 
related to foundational knowledge in literacy. 
-5days to refine within Literacy Coach training 
sessions (3 hrs each) , focused on refinement 
and integration: Word Identification; Self 
Questioning; Visual Imagery; LINCS Strategy & 
Routines; Fundamentals of Sentence Writing; 
and Paraphrasing ; Inference Strategy; 
Fundamentals; Proficiency; and Paragraph 
Writing 

-Additional motivation strategy Possible 
Selves, and interest inventories for adolescent 
-Incorporated resources for word mapping to 
support LINCs and DISSECT 

-Incorporated Comma Strategy to support 
Fundamentals and Proficiency of Sentence 
Writing. Site based job embedded 

-Literacy Coach included in school wide 
training 

-Engaged in practicum courses to support their 
coaching and specialist role 


School-based Coach Mentoring: 

In-class monitoring from certified trainers and 

CTL mentor coaches: 

- 3-4 days onsite coach mentoring — modeling, 
planning, observation/feedback by certified 
trainer and CTL mentor coach 

- Bi-monthly distance learning sessions 

- 2-3 days in class monitoring at school sites 
needing assistance by the certified trainer 

- Distance individual mentoring 

In-class mentoring/coaching by school 

literacy coach 

-Daily 

-Ongoing 

-Onsite and web-based support 


Literacy Coach and Administrator Knowledge, Attitudes, Beliefs, and Practices 


Curriculum and Materials: 

--High-interest adolescent reading materials, assessment 
materials, literary texts-novels, professional books, 
informational texts, content area texts 

--Technology integration, internet social networking, tablet 
computer 

--Learning Strategies Curriculum Manuals 

--Flip camera and web cam to support student engagement 
and collaboration 


Appendix A: Logic Model for the Targeted Intervention for Kentucky Content Literacy Consortium — Implementation Year 4 (2009-2010) 


CLASSROOM/SCHOOL PRACTICES: INTERMEDIATE 


--Provide 50- 60 minute targeted intervention class 
to no more than 15 targeted students in 6" and9" 
grade, scoring 2 years or more behind in reading ; 
Continue with strategies: DISSECT; ASKIT, SCENE, 
RAP, LINCS Routine using the following 7 steps: 1) 
Pretest and Make Commitments 2) Describe 3) 
Model 4) Verbal Practice 5) Controlled Practice 6) 
Advanced Practice 7) Posttest and Make 
Commitments through the following phases: 
Generalization; Orientation; Activation; Adaptation 
and Maintenance; Build strategy resources with 
paraphrasing and summarizing, INFER, PENS, and 
MARK. Promote motivation with Possible Selves 
and integrate technology - comma strategy. 


Administer diagnostics and pretest individual 
assessments to determine reading level and skills 
to tailor instruction and select specific instructional 
intervention. 


Application of job-embedded intervention 
strategies to support teacher skill in administering 
the routines learned from Professional 
Development training. 


Continuation of SIM™ routines in targeted 
intervention class every 4-6 weeks providing 
literacy instructional strategies. 


Use of CTL mentoring and instructional feedback 
to improve teacher implementation, and short- 
and long-term project planning 


Onsite coaching by certified trainer for targeted 
intervention teachers to develop leadership 
strategies and promote teacher implementation 
support. 


Assistance to utilize content relevant materials 


while implementing SIM routines and delivery 
options for flexible grouping. 


12 high schools, 11 middle schools, 24 Literacy Coaches 


Completed 9 graduate hours from the University of Louisville in literacy through job embedded and professional development training 


/ Improved 
implementation 
; of strategies 


.5 FTE (Full-Time Equivalency) — Literacy Coach to implement targeted intervention; .5 FTE — Literacy Coach to implement school-wide model 


TEACHER OUTCOMES 
SHORT-TERM LONGER-TERM 


Acquire and 
implement 
literacy 


Teacher efficacy 


STUDENT OUTCOMES 
SHORT-TERM LONGER-TERM 


Increase the use of new 
strategies for word 
identification, 
comprehension, vocabulary 
and fluency, and diagnostic 
of reading needs with 
progress monitoring 


Improve student 
achievement in 
reading 


Improve 
motivation for 
reading 


Regular application and 
weaving of SIM 
routines strategies 


Practice and apply within 
the intervention 
classroom; generalizing 
of targeted strategies 
outside the classroom in 
content areas; networking 
and supporting middle 
and high school levels 


Improve the use of 
reading strategies 
application 


Kathy Belcher, Project Director 

Latricia P. Bronger, Department of Special Education 
University of Louisville/SIM Professional Developer 

Dr. Brenda J. Overturf, Director of the Reading Program 
University of Louisville 


Appendix B: Logic Model for the Collaborative for Teaching and Learning Adolescent Literacy Model — Implementation Year 4 (2009-2010) 


PROGRAM INPUTS/ACTIVITIES 


Teacher Professional Development: 
Kentucky Striving Readers Conference —a 
1-day project-wide conference with over 
60 sessions offered to participating 
schools, provided by CTL mentors and 
school literacy coaches 

Literacy Technologies Training — 2 days 
per school 


Literacy Coach Professional 

Development: 

- 2-day coaching training — school wide 
model and coach training 

- 3 bi-monthly coaching training sessions, 
focused on skill development, reflection, 
and program planning 

- 1 data analysis training day 


School-based Coach Mentoring: 

- 5-7 days of onsite coach mentoring — 
modeling, planning, observation, and 
feedback 

- Weekly distance individual mentoring 

- Additional onsite and distance support 
for site-specific issues 


Leadership Training: 

Administrators 

- Participated in project conference 

- 1 day administrative leadership training 

- 1 project data analysis day 

Teacher Leadership Teams 

- Met monthly to support teachers 
implementation and program planning 


Teacher, Literacy Coach, and Administrator Knowledge, Attitudes, Beliefs, and Practices 


CLASSROOM/SCHOOL PRACTICES: 


INTERMEDIATE OUTCOMES 


Use of content literacy instructional 
strategies, as specified (six sub-domains, 
integrated with daily content 
instruction) 


Use of content literacy planning tools 
linked to Kentucky Core Content and 
Program of Studies 


Application of job-embedded coaching 
strategies to support teacher learning 
and application of strategies in service of 
learning 


Use of mentoring and instructional 
feedback to improve teacher 
implementation, and short- and long- 
term project planning 


Use of administrative and distributed 
leadership strategies and tools to 
monitor and promote teacher 
implementation, provide coaching 
support, and build capacity for gradually 
released implementation 


STUDENT OUTCOMES 


SHORT-TERM LONGER-TERM 


Improved 
self- efficacy 


Increased 
reading 

strategies 
used 


Increased 
reading 
motivation 


Improved 
engagement 
in learning 


va 


Increased 
reading 


Regular 
application of 
literacy 
strategies 


achievement 


The Collaborative for Teaching and Learning 


2008 
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Appendix C 
Summary of HLM Model Results for Student Achievement 


Exhibit Table 1 
6th Grade Student Achievement, NCE Score:-Summary of Model Results 


Fixed Effects 


Standard 

Effect Estimate Error df t value Pr>|t| 
Intercept 25.612 25.7484 6 0.99 0.3583 
School: KCCT reading scores, 

base year (spring 2006) 0.00283 0.0843 1116 0.03 0.9732 
School: Percent white students -0.1301 0.1975 1116 -0.66 0.5103 
School: Percent black students -0.3111 0.2891 1116 -1.08 0.2821 
School: Percent free/reduced -0.6492 0.0785 1116 -0.83 0.4086 
School: Percent with disabilities 0.3325 0.2936 1116 1.13 0.2576 
School: Average enrollment -0.0003 0.0036 1116 -0.08 0.9346 
Student: Fall GRADE score 0.6232 0.0416 1116 14.98 <.0001 
Student: Intervention class 1.0669 0.7162 1116 1.49 0.1366 
Student: Gender, male -0.1270 0.7336 1116 -0.17 0.8625 
Student: Ethnicity, minority -1.4492 1.1607 1116 -1.25 0.2122 
Student: SES, free/reduced lunch -1.3270 0.7930 1116 -1.67 0.0945 
Student: Special Education -4.2625 0.8483 1116 -5.02 <.0001 

Random Effects 
Variance Component Estimate ICC 
Level 2 — School 1.416 .001 
Level 1 — Student 144.09 
Random Effects (From Unconditional Model’) 

Variance Components Variance ICC 
Level 2 -School 2.126 .011 
Level 1 — Student 191.23 (24.7% level 1 variance explained by covariates) 


a) The unconditional model is a two-level model with students (level-1) nested in schools (level-2) and only an intercept term on 
the right-hand side of the model. 
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Exhibit Table 2 
6th Grade Student Achievement, NCE Score: Summary of Model Results — Non-Special Education Students 


Fixed Effects 


Standard 

Effect Estimate Error df t value Pr>|t| 
Intercept 66.4292 30.0631 6 2.21 0.0692 
School: KCCT reading scores, 

base year (spring 2006) 0.02050 0.09611 765 0.21 0.8312 
School: Percent white students -0.4768 0.2450 765 -1.95 0.0520 
School: Percent black students -0.7747 0.3494 765 -2.22 0.0269 
School: Percent free/reduced -0.06176 0.08324 765 -0.74 0.4584 
School: Percent with disabilities -0.09163 0.3100 765 -0.30 0.7677 
School: Average enrollment -0.00073 0.003969 765 -0.18 0.8537 
Student: Fall GRADE score 0.6309 0.05344 765 11.81 <.0001 
Student: Intervention class 1.0210 0.8642 765 1.18 0.2378 
Student: Gender, male 0.1182 0.8828 765 0.13 0.8935 
Student: Ethnicity, minority -3.0984 1.4533 765 -2.13 0.0333 
Student: SES, free/reduced lunch -0.7517 0.9457 765 -0.79 0.4269 
Random Effects 
Variance Component Estimate ICC 
Level 2 — School 0.789 0.005 


Level 1 — Student 145.50 
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Exhibit Table 3 
6th Grade Student Achievement, NCE Score: Summary of Model Results — Special Education Students 


Fixed Effects 


Effect Estimate SE df t value Pr>|t| 


Intercept -12.6552 34.5139 5 -0.37 0.7289 


School: KCCT reading scores, 


base year (spring 2006) 0.04754 0.1088 335 0.44 0.6624 
School: Percent white students 0.1903 0.2633 335 0.72 0.4702 
School: Percent black students 0.1457 0.4317 335 0.34 0.7360 
School: Percent free/reduced -0.07664 0.09809 335 -0.78 0.4352 
School: Percent with disabilities 0.8502 0.3672 335 2.32 0.0212 
School: Average enrollment 0.000719 0.004401 335 0.16 0.8702 
Student: Fall GRADE score 0.5933 0.06657 335 8.91 <.0001 
Student: Intervention class 1.2622 1.2907 335 0.98 0.3288 
Student: Gender, male -1.1440 1.3529 335 -0.85 0.3984 
Student: Ethnicity, minority -1.7941 1.9867 335 -0.90 0.3671 
Student: SES, free/reduced lunch 2.5714 1.4459 335 1.78 0.0762 


Random Effects 


Variance Component Estimate ICC 
Level 2 — School 0 0 
Level 1 — Student 140.50 
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Exhibit Table 4 
9th Grade Student Achievement, NCE Score:-Summary of Model Results 


Fixed Effects 


Standard 

Effect Estimate Error df t value Pr>|t| 
Intercept 34.3669 48.4645 5 0.71 0.5099 
School: KCCT reading scores, 

base year (spring 2006) 0.0545 0.383 1109 0.14 0.8870 
School: Percent white students -0.3265 0.567 1109 -0.58 0.5653 
School: Percent black students -0.6074 0.503 1109 -1.21 0.2273 
School: Percent free/reduced 0.0476 0.249 1109 0.19 0.8488 
School: Percent with disabilities 1.0117 1.625 1109 0.62 0.5338 
School: Average enrollment -0.0074 0.017 1109 -0.43 0.6676 
Student: Fall NCE scores 0.6283 0.049 1109 12.62 <.0001 
Student: Intervention class 1.6852 0.785 1109 2.15 0.0321 
Student: Gender, male -0.1419 0.8032 1109 -0.18 0.8598 
Student: Ethnicity, minority -2.0685 1.375 1109 -1.50 0.1328 
Student: SES, free/reduced lunch -0.4319 0.8407 1109 -0.51 0.6075 
Student: Special Education -4.4435 0.932 1109 -4.76 <.0001 

Random Effects 
Variance Components Variance ICC 
Level 2 — School 3.094 .018 
Level 1 — Student 171.34 
Random Effects (From Unconditional Model’) 

Variance Components Variance ICC 
Level 2 -School 0 0 
Level 1 — Student 207.37 (17.4% level 1 variance explained by covariates) 


a) The unconditional model is a two-level model with students (level-1) nested in schools (level-2) and only an intercept term on 
the right-hand side of the model. 
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Exhibit Table 5 
9th Grade Student Achievement, NCE Score: Summary of Model Results — Non-Special Education Students 


Fixed Effects 


Standard 

Effect Estimate Error df t value Pr>|t| 
Intercept 6.2524 49.5195 5 0.13 0.9044 
School: KCCT reading scores, 

base year (spring 2006) 0.08367 0.3593 810 0.23 0.8159 
School: Percent white students 0.03744 0.5761 810 0.06 0.9482 
School: Percent black students -0.07687 0.5459 810 -0.14 0.8880 
School: Percent free/reduced -0.01057 0.2301 810 -0.05 0.9634 
School: Percent with disabilities 0.7299 1.5299 810 0.48 0.6334 
School: Average enrollment -0.00570 0.01609 810 -0.35 0.7233 
Student: Fall GRADE score 0.5876 0.05971 810 9.84 <.0001 
Student: Intervention class 2.6839 0.8852 810 3.03 0.0025 
Student: Gender, male 0.3322 0.9058 810 0.37 0.7139 
Student: Ethnicity, minority -2.2698 1.5783 810 -1.44 0.1508 
Student: SES, free/reduced lunch -0.1293 0.9298 810 -0.14 0.8895 
Random Effects 
Variance Component Estimate ICC 
Level 2 — School 1.416 .010 


Level 1 — Student 144.09 
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9th Grade Student Achievement, NCE Score: Summary of Model Results — Special Education Students 


Fixed Effects 


Standard 

Effect Estimate Error df t value Pr>|t| 
Intercept 70.6668 84.1211 4 0.84 0.4482 
School: KCCT reading scores, 

base year (spring 2006) -0.2418 0.7027 284 -0.34 0.7311 
School: Percent white students -0.6600 1.0006 284 -0.66 0.5101 
School: Percent black students -1.2580 0.8620 284 -1.46 0.1456 
School: Percent free/reduced 0.3271 0.4607 284 0.71 0.4782 
School: Percent with disabilities 2.1923 2.9246 284 0.75 0.4541 
School: Average enrollment -0.01583 0.03114 284 -0.51 0.6115 
Student: Fall GRADE score 0.7163 0.09546 284 7.50 <.0001 
Student: Intervention class -1.0123 1.6891 284 -0.60 0.5494 
Student: Gender, male 1.2653 1.7708 284 0.71 0.4755 
Student: Ethnicity, minority -2.0521 2.8193 284 -0.73 0.4673 
Student: SES, free/reduced lunch -1.4142 1.9120 284 -0.74 0.4601 
Random Effects 
Variance Component Estimate ICC 
Level 2 — School 9.0567 0.043 
Level 1 — Student 200.41 


Cantrell, Almasi, Carter, & Rintamaa (2011) 


Striving Readers Final Evaluation Report: Danville, Kentucky 


Exhibit Table 7 
6th Grade Student Reading Strategy Use; MARS! Scores: - Summary of Model Results 


Fixed Effects 


Standard 

Effect Estimate Error df t value Pr>|t| 
Intercept 4.1069 1.428 7 2.87 0.0238 
School: KCCT reading scores, 

base year (spring 2006) 0.0073 0.004 809 1.72 0.0866 
School: Percent white students -0.0217 0.0121 809 -1.79 0.0743 
School: Percent black students -0.0381 0.018 809 -2.14 0.0323 
School: Percent free/reduced lunch -0.0016 0.004 809 -0.44 0.6589 
School: Percent with disabilities -0.0231 0.014 809 -1.61 0.1075 
School: Average enrollment -0.0002 0.0001 809 -0.97 0.3337 
Student: Fall Marsi score 0.3526 0.034 809 10.25 <.0001 
Student: Intervention class 0.0857 0.044 809 1.91 0.0565 
Student: Gender, male -0.0954 0.045 809 -2.08 0.0379 
Student: Ethnicity, minority 0.1058 0.071 809 1.48 0.1382 
Student: SES, free/reduced lunch 0.0984 0.048 809 2.02 0.0439 
Student: Special Education 0.0382 0.052 809 0.75 0.4552 


Random Effects 


Variance Components Estimate ICC 
Level 2 — School 0.0000 0.000 
Level 1 — Student 0.414 
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Exhibit Table 8 
9th Grade Student Reading Strategy Use; MARS! Scores: - Summary of Model Results 


Fixed Effects 


Standard 

Effect Estimate Error df t value Pr>|t| 
Intercept 2.8309 2.994 5 0.95 0.3878 
School: KCCT reading scores, -0.0196 0.024 691 -0.8 0.4159 

base year (spring 2006) 
School: Percent white students -0.0049 0.035 691 -0.14 0.8897 
School: Percent black students -0.0164 0.031 691 -0.52 0.6009 
School: Percent free/reduced lunch 0.0220 0.016 691 1.14 0.1584 
School: Percent with disabilities -0.0196 0.024 691 -0.81 0.4159 
School: Average enrollment -0.003 0.001 691 -0.29 0.7749 
Student: Fall MARSI score 0.4287 0.038 691 11.20 <.0001 
Student: Intervention student 0.0682 0.051 691 1.32 0.1860 
Student: Gender, male -0.1704 0.053 691 -3.20 0.0015 
Student: Ethnicity, minority 0.2664 0.084 691 3.13 0.0018 
Student: SES, free/reduced lunch -0.1138 0.054 691 -2.08 0.0378 
Student: Special Education -0.0112 0.062 691 -0.18 0.8557 
Random Effects 
Variance Components Estimate ICC 
Level 2 — School 0.010 0.022 


Level 1 — Student 0.462 
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Exhibit Table 9 
6th Grade Student Motivation; MRQ Scores: - Summary of Model Results 


Fixed Effects 


Effect Estimate SE df t value Pr>|t| 


Intercept 3.1504 1.305 7 2.41 0.0466 


School: KCCT reading scores, 


base year (spring 2006) 0.0051 0.004 807 1.25 0.2107 
School: Percent white students -0.017 0.010 807 -1.64 0.1009 
School: Percent black students -0.0253 0.015 807 -1.68 0.0938 
School: Percent free/reduced lunch 0.0036 0.004 807 0.95 0.3421 
School: Percent with disabilities -0.0232 0.014 807 -1.60 0.1092 
School: Average enrollment 8.913 0.0001 807 0.05 0.9607 
Student: Fall MRQ scores 0.3571 0.034 807 10.41 <.0001 
Student: Intervention class 0.0746 0.030 807 2.42 0.0156 
Student: Gender, male -0.0391 0.031 807 -1.23 0.2181 
Student: Ethnicity, minority 0.0530 0.049 807 1.08 0.2827 
Student: SES, free/reduced lunch 0.0333 0.0336 807 0.99 0.3221 
Student: Special Education -0.0127 0.035 807 -0.36 0.7207 


Random Effects 


Variance Components Estimate ICC 
Level 2 -School 0.004 0.019 
Level 1 — Student 0.195 
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Exhibit Table 10 
9th Grade Student Motivation; MRQ Scores: - Summary of Model Results 


Fixed Effects 


Effect Estimate SE df t value Pr>|t| 


Intercept 2.0876 1.465 5 1.42 0.2137 


School: KCCT reading scores, 


base year (spring 2006) -0.0155 0.046 689 -0.33 0.7401 
School: Percent white students -0.0078 0.017 689 -0.46 0.6486 
School: Percent black students -0.0041 0.016 689 -0.25 0.8035 
School: Percent free/reduced lunch 0.0111 0.007 689 1.60 0.1110 
School: Percent with disabilities -0.0155 0.047 689 -0.33 0.7401 
School: Average enrollment 0.0002 0.0004 689 0.40 0.6880 
Student: Fall MRQ scores 0.1502 0.038 689 13.34 <.0001 
Student: Intervention class 0.1193 0.034 689 3.45 0.0006 
Student: Gender, male -0.1193 0.035 689 -3.35 0.0008 
Student: Ethnicity, minority 0.1346 0.057 689 2.36 0.0187 
Student: SES, free/reduced lunch -0.0677 0.036 689 -1.85 0.0642 
Student: Special Education -0.0713 0.041 689 

-1.71 0.0869 


Random Effects 


Variance Components Estimate ICC 
Level 2 — School 0.000 0.000 
Level 1 — Student 0.209 
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APPENDIX D 
School KCCT Scores for Striving Readers and Matched Schools 


Figure D1 
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Striving Readers Middle School Writing Scores 
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Appendix E 
Measures 


COLLABORATIVE CENTER FOR 


\ 


Student Reading Strategies 
DEVELOPMENT Inventory 


Listed below are statements about what people do when they read academic or school-related 
materials such as textbooks or library books. There are no right or wrong answers. Please answer 
as honestly and truthfully as you can and provide an answer for each question. 


Your answers on this survey are completely confidential. 


Information about this survey will be released in summary form only. 


1 = 1 never or almost never do this 

2 = 1 do this only occasionally 

3 =I! sometimes do this (about 50% of the time) 
4 =1 usually do this 

5 = | always or almost always do this 


1. | have a purpose in mind when | read. 

2. | take notes while reading to help me understand what | read. 

3. | think about what | know to help me understand what | read. 

A. | preview the text to see what it’s about before reading it. 

5. When text becomes difficult, | read aloud to help me understand what | 
read. 

6. | summarize what | read to reflect on important information in the text. 

7. | think about whether the content of the text fits my reading purpose. 

8. | read slowly but carefully to be sure | understand what I’m reading. 

9. | discuss what | read with others to check my understanding. 

10. | skim the text first by noting characteristics like length and organization. 

11. | try to get back on track when | lose concentration. 
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12. 


13. 


14. 


15. 


16. 


17. 


18. 


19. 


20. 


21. 


22. 


23. 


24. 


20. 


26. 


27. 


28. 


oo 


30. 
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| underline or circle information in the text to help me remember it. 


| adjust my reading speed according to what I’m reading. 


1 = 1 never or almost never do this 
2 =I! do this only occasionally 
3 =! sometimes do this (about 50% of the time) 
4 = 1 usually do this 
5 = | always or almost always do this 
| decide what to read closely and what to ignore. 


| use reference materials such as dictionaries to help me understand 
what | read. 


When text becomes difficult, | pay closer attention to what I’m reading. 
| use tables, figures, and pictures in text to increase my understanding. 
| stop from time to time and think about what I’m reading. 

| use context clues to help me better understand what I’m reading. 


| paraphrase (restate ideas in my own words) to better understand what 
| read. 


| try to picture or visualize information to help remember what | read. 


| use typographical aids like boldface, and italics to identify key 
information. 


| critically analyze and evaluate the information presented in the text. 
| go back and forth in the text to find relationships among ideas in it. 

| check my understanding when | come across conflicting information. 
| try to guess what the material is about when | read. 

When text becomes difficult, | reread to increase my understanding. 

| ask myself questions | like to have answered in the text. 

| check to see if my guesses about the text are right or wrong. 


| try to guess the meaning of unknown words or phrases. 
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oe. 
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Please mark your responses to these statements on the 
scantron sheet: 


| would rate my overall reading ability as: 
1) excellent 2) average 3) notso good 


| would rate my overall academic performance in school as: 
1) excellent 2) average 3) notso good 
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Motivation for Reading Questionnaire for Adolescents*, (Cantrell, Almasi, & Rintamaa © 2006) p.3. 


paeredersmnions Sie 1) Striving Readers Motivation for Reading 
Questionnaire for Adolescents* 


DEVELOPMENT 
1 = Strongly Disagree 

2 = Disagree 

3 = Agree 

4 = Strongly Agree 
33. | visit the library often with friends or family. 
34. | like hard, challenging books. 
35. | Know that | will do well reading in my classes next year. 
36. If the teacher discusses something interesting | might read more about it. 
37. | like it when the questions in books make me think. 
38. | read about my hobbies to learn more about them. 
39. | am a good reader. 
40. | enjoy reading magazines. 
41. | often read to other people. 
42. | like being the only one who knows an answer in something we read. 
43. | read to learn new information about topics that interest me. 


44. My friends sometimes tell me | am a good reader. 


45. | learn more from reading than most students in the class. 
46. | like to read about new things. 

47. | like hearing the teacher say | read well. 

48. | sometimes read to my parents. 


49. My friends and | like to trade things to read. 


1 = Strongly Disagree 
2 = Disagree 

3 = Agree 

4 = Strongly Agree 


Cantrell, Almasi, Carter, & Rintamaa (2011) 


50. 


51. 


52. 


53. 


54. 


55. 


56. 


57. 


58. 


59. 


60. 


61. 


62. 


63. 


64. 


65. 


66. 


67. 


68. 


69. 


70. 


71. 
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| don’t like reading something when the words are too difficult. 
| make pictures in my mind when | read. 
| always read exactly as the teacher wants. 
| usually learn difficult things by reading. 
| don’t like vocabulary questions. 
Complicated texts are no fun to read. 
| am happy when someone recognizes my reading. 
| feel like | make friends with people in good books. 
My parents often tell me what a good job | am doing with reading. 
Finishing every reading assignment is very important to me. 
| talk to my friends about what | am reading. 
If 1am reading an interesting topic | sometimes lose track of time. 
| like to get compliments for my reading. 
Grades are a good way to see how well you read. 
| like to help my friends with the reading we do for school. 
| read to improve my grades. 
| enjoy a long, involved story or fiction book. 
| like to tell my family about what | am reading. 
| try to get more answers right than friends. 
If the project is interesting, | can read difficult material. 
1 = Strongly Disagree 
2 = Disagree 
3 = Agree 
4 = Strongly Agree 
| enjoy reading books about people in different countries. 


| enjoy searching for information on the internet. 
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72. 


73. 


74. 


15. 


76. 


77. 


78. 


19. 


80. 


81. 


82. 
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| always try to finish my reading on time. 

If a book is interesting, | don’t care how hard it is to read. 

| like to finish my reading before other students. 

In comparison to my other school work | am best at reading. 

| am willing to work hard to read better than my friends. 

| don’t like it when there are too many new ideas in the text. 

It is very important to me to be a good reader. 

In comparison to other activities | do, it is very important to me to be a good reader. 
| am avery good reader. 

| put forth my best effort on this survey. 


This survey was easy for me. 


Thank you for taking time for to fill out this survey! 


*based on the Motivation for Reading Questionnaire contained in: 
Baker, L., & Wigfield, A. (1999). Dimensions of children’s motivation for reading and their relations to reading activity 
and reading achievement. Reading Research Quarterly, 34, 452-477. 
Wigfield, A., & Guthrie, J. (1997). Relations of children’s motivation for reading to the amount and breadth of 
their reading. Journal of Educational Psychology, 89, 420-432. 
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COLLABORATIVE CENTER FOR 


Vv 


DEVELOPMENT 


Striving Readers Teacher Survey* 

Name (please print) Date 

School 

E-mail address (school) 

Directions: Please completely fill in each circle. 
Thank you for completing this questionnaire. To ensure your privacy, information from this 
survey will be released in summary form only. 

PARTI. Teacher Background 
1. What subject(s) do you currently teach? (include all) 

O Language Arts/Speech O Science O Social Studies 


O Special Ed O Reading O Mathematics 


O Administrator (If you are an Administrator, please fill in circle and skip to Part Il) 
O Other 


2. What subject(s) is your primary teaching responsibility? 
O Language Arts/Speech =O Science O Social Studies O Mathematics 
O Special Ed O Reading O Other 


3. What is your gender:O male  O female 


4, Ethnicity (optional): O Caucasian O African American O Hispanic O Asian American 
O Native Americans O Other 


5. How many years of teaching experience do you have? 
O0-3years O4-7 years O 8-11 years O 12-15 years O 16 years and 


mp 


6. What grade level(s) do you teach in school? (include all) 
O6th O7th O8th O9th O10th O 11th O 12th O Other 
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7. What is the highest degree you have earned? 
O Bachelor’s O Master’s O Specialist/Rank I O Doctorate 


8. How many years ago did you receive your highest academic degree? 
OO0-3years O4-7 years O 8-11 years O 12-15 years O 16 years and 


uP 


9. How many years have you been teaching in your present school? 
O0-3years O4-7 years O 8-11 years O 12-15 years O 16 years and 


up 
10. Are you presently teaching under emergency certification? O yes Ono 
11. Have you had other types of literacy-focused professional development? O yes Ono 


12. When were you first hired as a teacher at your school? Month Year 


13. What is your area of certification? (include all)_ 
O Language Arts O Science O Social Studies 
O Special/Speech Ed O Reading O Mathematics 


PART II. Teacher Beliefs 
This survey focuses on reading, which may be defined as constructing meaning from a variety of texts, 
including books, charts, graphs, technology, etc. 


Please use the scale below to answer the questions that follow. Please provide a response to every question. 
If none of the alternatives provided for a question corresponds exactly to your position or opinion, select the 
alternative that comes closest to the answer you would like to give. If you teach more than one subject 
area, please think about the subject that is your primary responsibility when answering the questions. 


1=Strongly Agree 
2=Moderately Agree 
3=Agree slightly more than disagree 
4=Disagree slightly more than agree 
5=Moderately Disagree 
6=Strongly Disagree 


Agr Dis 

ee agree 

slightly slightly 
mor 

e than more 

than 

Disagree — 


[ai | Factors beyond my entolhave| OO of of 6, 0] 


Stro 
ngly 


Disagree 
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Agr Dis 
ee agree 


slightly oe 


ngly 
Disagree 


z 
a 


Iam good at helping all the 
students in my classes make 
significant improvement in their 
reading comprehension. 

Some students are not going to 
make a lot of progress this year in 
reading, no matter what I do. 

I am certain that [am making a 
difference in the lives of my 
students when it comes to reading. 

There is little I can do to ensure 
that all my students make 
significant progress in reading this 


36. I can deal with almost any 
reading problem. 

37. The amount a student can read 
is primarily related to family 


background. 


38. If students are not willing to 
read, I can do little about it. 
If parents would do more for 
their children’s reading, I could do 


If one of my students could not 
do a class assignment, I would be 
able to accurately assess whether 
the assignment was at the correct 
level of reading difficulty. 


O 
4 
O 
4 
O 
4 
O 
4 
O 
4 
O 
4 
O 
4 
O 
4 
O 
4 


O 
1 
O 
1 
O 
1 
O 
1 
O 
1 
O 
1 


O O 
2 3 
O O 
2 3 
O O 
2 3 
O O 
2 3 
O O 
2 3 
O O 
2 3 


If I really try hard, I can get 
through to even the most difficult 
or unmotivated students. 

When it comes right down to it, 
I really cannot do much about a 
student’s reading because most of a 
student’s performance depends 


wo [oe 
wo foo 
ir 
mr 


upon his or her home environment. 
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Agr Dis 

ee agree 

slightly | slightly 
mor 


Stro 


ngly 


e than more ; 
Disagree 


than 
Disagree Agr 
ee 

When a student does better 
than usual in reading, it is often 
because I exerted a little extra effort. 

I know how to teach 
vocabulary effectively. Vocabulary 
refers to the understanding of word 
meanings. 

Even when I try very hard, I do 
not teach reading as well as Ido 
most subjects. 

When my students’ reading 
improves, it is most often due to my 
having found a more effective 
teaching approach. 

I know the steps necessary to 
teach reading in my content area 


effectively. 


Iam convinced that I am able to 
successfully teach all relevant 
subject content to even the most 
difficult students. 

I effectively use grouping to 
engage students in reading in my 
content area. 

Iam not very effective in 
monitoring students’ reading 
ability. 

If students are underachieving 
in reading, it is most likely due to 
my ineffective teaching. 


I generally deal with students’ 

Even when I try very hard, I do 
not teach writing as well as I teach 
most subjects. 

My good teaching can 
overcome the inadequacy of a 


student's reading. 


a I should not be held responsible 


Fs Fe sg 
sd soi i at st en a 
i 
dae ol eo oft wa 
i a a a a a 
Ps 
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Agr Dis 
ee agree 
slightly | slightly one 
mor . 1 
e than more re 
than 
Disagree 


5. | for the low reading achievement of 
some students. 


When a low-achieving student 
progresses in reading, it is usually 
due to my extra attention. 

I understand reading concepts 
well enough to be effective in 
teaching it along with content area 
material. 

Increased effort in teaching 
reading will produce little change 
in some students’ achievement in 
my class. 

Iam generally responsible for 


the reading achievement of 
students in my class. 


Students’ achievement in a 
subject is directly related to my 
effectiveness in teaching that 
content area AND my ability to 
teach reading. 
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Agr Dis 

ee agree 

slightly | slightly 
mor 


Stro 


ngly 


e than more ; 
Disagree 


than 
Disagree 


If parents comment that their 
child is showing more interest in a 
subject at school, it is probably due 
to my performance. 

I know how to teach my 
students to decode unknown words 
they read in my content area. 
Decoding refers to the method or 
strategy a student uses to “figure 
out” a word. 


I wonder if I have the necessary 
skills to teach reading. 


My effectiveness in teaching 


reading has little influence on 
students with low motivation. 


Given a choice, I would not 
5. | invite someone in to evaluate my 


teaching of reading within my 
content area. 

When a student has difficulty 
understanding his/her reading, Iam 
usually at a loss as to how to help 
the student understand it better. 

I do not know what to do to get 
students excited about reading in 
my content area. 

I am certain that I know how to 
enhance students’ reading fluency 
in my content area. (Fluency refers 
to the ability to read text accurately 
and quickly and with expression). 

Even if I teach the content area 
well, I cannot help some kids to 
read better. 

I believe my students’ prior 
experiences will directly affect their 
ability to understand what they are 
reading in my content area. 


I believe I teach students to O O O O O 
question the viewpoint of text 1 2 3 4 5 
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If a child does not learn 
something the first time I will try 
another way. 


Most teachers in my school are 
skilled in various methods of 
teaching reading. 

Most teachers in my school are 
well prepared to teach the subjects 
they are assigned to teach. 

Most teachers in my school 
really believe every child can learn 
to read in their content area. 

If a child does not want to read 
in their content area, most teachers 
in my school give up. 

Most teachers in my school do 
not have the skills needed to 
produce meaningful student 
learning. 

Most teachers in my school 
have what it takes to get the 
children to read in their content 
area. 

Most teachers in my school are 
able to get through to struggling 
readers. 

Most teachers in my school are 
confident they will be able to 
motivate their struggling readers. 

The lack of instructional 
materials and supplies makes 
teaching very difficult. 

Most teachers in my school do 
not have the skills to deal with 
student disciplinary problems. 

Most teachers in my school 
think there are some students that 
no one can reach. 
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ee agree 
slightly slightly 
mor 
ngly 


e than more ; 
Disagree 


than 
Disagree Agr 


ee 
O O 
3 4 
O O 
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The quality of school facilities 


in my school really facilitates the 


teaching and learning process for 
reading in the content area. 

Home life provides so many 
advantages to students in my 
school that they are bound to learn. 

The students in my school come 
to school ready to learn. 


agree 
slightly slightly 


mor 
ngly 


e than more ; 
Disagree 


than 
Disagree 


Drug and alcohol abuse in my 
school’s community make learning 
difficult for students. 

The opportunities in my 
school’s community help ensure 
that the students in my school will 
learn. 

Students in my school just are 
not motivated to learn. 

Learning is more difficult at my 
school because students are worried 
about their safety. 


Thank you for completing this survey! 


* Adapted from: 

Woolfolk, A. E., & Hoy, W. K. (1990). Prospective teachers’ sense of efficacy and beliefs about control. Journal of Educational 
Psychology, 82, 81-91. 

Hoy, W. K., & Woolfolk, A. E. (1993). Teachers’ sense of efficacy and the organizational health of schools. The Elementary School Journal, 93, 
356-372. 
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Appendix F 
Grantee Administered Implementation Questionnaire Results 
Kathy Belcher, Project Director, Danville Independent Schools 
Surveys of literacy coach and administrators’ perceptions of the Striving Readers program 

during the previous year were administered to the participants within the intervention schools. 
Surveys were administered in Year 3 from May to June 2009 through an online survey system. 
Each literacy coach and administrator within the intervention schools was emailed a request to 
participate by the Project Director. These surveys were used to examine the perceived impact of 
the program, implementation of the program, additional needs not addressed within the program, 
and the likelihood for the schools to participate in all or parts of the program in the future. Survey 
items addressed the school-wide literacy and the intervention models within the participating 
schools. The Literacy Coach Implementation Questionnaire consisted of 13 items and the 
Administrator Implementation Questionnaire consisted of 17 items. Both surveys utilized open- 
ended and closed-ended items. The majority of the items included a list of response options with 
instructions to check all that apply, check only one response option, or to rate each response on a 
one to four scale. A few items instructed the participants to provide open-ended responses and 
some of the closed-ended response items allowed participants to provide comments or asked 
participants to explain their responses. Only closed-ended response options were included in 


these results. 


Table F1 
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Administrator and Literacy Coach Implementation Questionnaire Results 


Item 


Responses 


Implementation 


At this time, what percentage 
of your staff is implementing the 
school-wide literacy strategies in 
your school this year on a regular 
basis (i.e. daily/weekly)? 


In which content area do 
you see the greatest 
implementation in your school? 


In which content area do 
you see the least 
implementation in your school? 


Concerning the 
implementation levels of the 
content literacy strategies at your 
school, please rate the following 
content areas on a scale from 1-4 
(left to right) with 4 being high 
implementation in your school 
and 1 being the area with the 
lowest level of implementation. 


How would you rate the 
quality of the literacy coaching 
taking place in your school? 


Item 


Administrator 


Literacy Coach 


Twenty-five percent (3.6%) 
Fifty percent (10.7%) 
Seventy-five percent (64.3%) 
One hundred percent (21.4%) 


Language Arts (42.9%) 
Science (25%) 

Math (21.4%) 

Social Studies (10.7%) 
Arts and Humanities (0%) 
Health/PE (0%) 


Language Arts (14.8%) 
Science (0%) 

Math (18.5%) 

Social Studies (3.7%) 

Arts and Humanities (18.5%) 
Health/PE (44.4%) 


Language Arts (M = 3.07) 
Science (M = 2.93) 

Math (M = 2.75) 

Social Studies (M = 2.89) 


Arts and Humanities (M = 
2.46) 


Health/PE (M = 1.93) 


Poor (0%) 

Adequate (21.4%) 
Above Average (28.6%) 
Excellent (50%) 


Twenty-five percent (4.2%) 
Fifty percent (29.2%) 
Seventy-five percent (58.3%) 
One hundred percent (8.3%) 


Language Arts (25%) 
Science (33.3%) 

Math (12.5%) 

Social Studies (20.8%) 

Arts and Humanities (8.3%) 
Health/PE (0%) 


Language Arts (16.7%) 
Science (4.2%) 

Math (25%) 

Social Studies (8.3%) 

Arts and Humanities (8.3%) 
Health/PE (37.5%) 


Language Arts (M = 2.91) 
Science (M = 3.04) 

Math (M = 2.33) 

Social Studies (M = 2.83) 


Arts and Humanities (M = 
2.52) 


Health/PE (M = 1.86) 


Responses 
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How would you rate the 
quality of instruction provided 
in the targeted intervention? 


On a rating scale of 1-4 with 
1 being ‘not supportive’ and 4 
being 'very supportive’, please 
rate the support of your school 
administrator in implementing 
the initiatives in the Striving 
Readers Project. 


Learning Outcomes 


To what extent do you 
believe the school-wide content 
literacy approaches are 
supporting improved student 
learning? 


Do you see evidence that the 
school-wide model is helping to 
improve student achievement? 


To what extent is the 
targeted intervention making a 
positive impact on student 
learning? 


Do you see the targeted 
intervention class making an 
impact on improving reading? 


On a rating scale from 1-4 
with 1 being ‘not effective’ and 4 
being ‘very effective’, please rate 
the overall effectiveness of the 
school-wide model in helping to 
increase student achievement? 


Item 


Administrator Literacy Coach 
Poor (0%) 
Adequate (7.1%) 
Above Average (32.1%) 
Excellent (60.7%) 
One (0%) 
Two (25%) 


Not (0%) 
Little (0%) 
Somewhat (42.9%) 


Three (16.7%) 
Four (58.3%) 


Substantially (57.1%) 


Not (0%) 
Little (7.1%) 
Somewhat (39.3%) 


Yes (95.8%) 
No (4.2%) 


Substantially (53.6%) 


Yes (91.7%) 
No (8.3%) 


One (0%) 

Two (16.7%) 
Three (50%) 
Four (33.3%) 


Responses 


Administrator 


Literacy Coach 
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How often do you use the 
Content Literacy Building 
Walkthrough Tool and/or 
Classroom Observation Tool to 
assess the progress of content 
literacy integration in your 
school? 


How often do you reassess 
your school's growth using the 
CTL Adolescent Literacy 
Performance Guide (lykert 
scale)? 


How often do you meet with 


your Literacy Coach to discuss 
general and specific content 
literacy and coaching needs in 
your school? 
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Seldom (25%) 
Somewhat (14.3%) 


Often (once per quarter; 
28.6%) 


Regularly (monthly; 32.1%) 


Monthly (7.1%) 
Quarterly (10.7%) 
Once a semester (32.1%) 


Annually (50%) 


Once per year (0%) 
Monthly (89.3%) 
Quarterly (10.7%) 
Never (0%) 


Perceived Needs and Sustainability 


What is needed to improve 


the school-wide content 


literacy model in your school? 


(Check all that are needed) 


If funds were available, 
would your school want to 
continue the Striving Readers 
initiatives for the 2010-2011 
school year? 


Item 


Continued PD for CLM 
(39.3%) 


Continued Coaching for 
CLM (64.3%) 


Classroom materials for the 
CLM (39.3%) 


Technology used to teach 
instructional strategies (60.7%) 


More intentional 
implementation on the part of the 
teaching staff (67.9%) 


Yes (87.5%) 
No (12.5%) 


Responses 


If funds are available to 


Administrator Literacy Coach 


Targeted Intervention class Targeted Intervention class 
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continue the Striving Readers 
initiatives for the year 2010-2011 
and your school decided to 
continue the efforts set by the 
Striving Readers grant, list the 
areas you would want to see 
continued. 


What is needed to improve 
the targeted intervention model 
in your school? (Check all 
that are needed) 


Item 
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(71.4%) 
School wide CLM (82.1%) 


Intervention teacher half time 
(57.1%) 


School wide Content 
Literacy Coach half time (67.9%) 


Two day school wide PD 
summer institutes (32.1%) 


Four day literacy coach PD 
summer institutes (25%) 


Monthly Literacy Coach 
PD/support (32.1%) 


Three administrator’s 
meetings (14.3%) 


Other (3.6%) 


Continued PD for the 
Intervention teachers (25%) 


Continued coaching for the 
Intervention teachers (39.3%) 


More classroom materials for 
the targeted intervention (25%) 


Technology used to teach 
instructional strategies (57.1%) 


More intentional 
implementation on the part of the 
Intervention teacher (17.9%) 


Other (10.7%) 


(58.3%) 
School wide CLM (75%) 


Intervention teacher half time 
(58.3%) 


School wide Content 
Literacy Coach half time (66.7%) 


Two day school wide PD 
summer institutes (54.2%) 


Four day literacy coach PD 
summer institutes (37.5%) 


Monthly Literacy Coach 
PD/support (50%) 


Three administrator’s 
meetings (20.8%) 


Other (8.3%) 


Responses 


Administrator 


Literacy Coach 
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What steps has your school 
taken to ensure the efforts made 
for the school-wide content 
literacy model and the targeted 
intervention will be sustained 
after the Striving Readers grant is 
completed. 
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My school principal is 
leading my staff and district to 
include using school-wide 
content literacy strategies and 
provide a targeted intervention as 
a major part of the school and 
district comprehensive plans. 
(41.7%) 


I believe my principal 
acknowledges there is positive 
evidence that the school-wide 
and targeted intervention models 
are needed for improvement of 
student achievement but have not 
set into place sustainability. 
(54.2%) 


My school principal sees 
evidence of improvement of 
student achievement, but only 
feels the school-wide content 
literacy model needs to be 
sustained. (4.2%) 


My school principal sees 
evidence of improvement of 
student achievement, but feels 
only the need to support 
sustaining the targeted 
intervention model. (0%) 


In my opinion, my school 
principal does not see evidence 
that the school-wide content 
literacy model nor the targeted 
intervention is overall effective 
and therefore probably will not 
support continuing efforts 
outlined in the Striving Readers 
initiatives. (8.3%) 
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